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Abstract 

The  purpose  of  this  study  was  to  determine  how 
Production  Readiness  Reviews  (PRR)  were  planned, 
implemented,  and  managed  by  selected  programs  within  Air 
Force  Systems  Command.  In  1983,  the  Defense  Science  Board 
was  tasked  with  reviewing  ways  to  improve  and  accelerate  the 
transition  of  weapon  systems  into  production.  During  1985, 
the  General  Accounting  Office  reviewed  why  some  weapon 
systems  encounter  production  problems  while  others  do  not. 
Both  studies  provided  recommendations  on  way3  to  improve  the 
transition  process.  However,  new  production  programs 
continue  to  suffer  significant  production  problems  which 
invariably  result  in  cost  increases  and  schedule  delays. 

This  study  reviewed  ten  programs  which  recently 
conducted  PRRs  in  an  attempt  to  look  at  actual  PRR 
practices  and  factors  which  influenced  the  selection  of 
these  practices.  In  addition,  an  attempt  was  made  to 
determine  whether  recommendations  from  the  DSB  and  GAO 
studies  were  implemented.  Included  in  this  study  were 
reviews  of  PRR  Plans,  PRR  Reports,  Contract  Statements  of 
Work,  and  PRR  briefings.  In  addition,  key  managers  were 
interviewed  for  their  views  of  the  PRR  process  and 
recommendations  for  changes. 

Based  on  the  data  from  this  study,  it  was  apparent 
that  all  ten  programs  met  the  basic  DOD  requirements  for  PRR 
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implementation.  However,  DOD  Regulations  and  Directives  did 
not  provide  sufficient  guidance  on  planning  and  managing  PRRs . 
The  result  was  several  different  planning  approaches  that 
were  generally  dependent  on  program  size,  cost,  and 
complexity.  The  degree  of  experience  and  expertise  of 
program  office  personnel  also  played  a  significant  role  in 
PRR  planning.  The  different  approaches  taken  and  the 
factors  influencing  these  approaches  are  provided  in  Chapter 
III.  Specific  conclusions  and  recommendations  are  provided 
in  Chapter  IV. 
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A  CASE  STUDY  OF  THE  PLANNING  AND  IMPLEMENTATION  OF 
PRODUCTION  READINESS  REVIEWS  FOR  SELECTED 
WEAPON  SYSTEMS  WITHIN  AIR  FORCE  SYSTEMS  COMMAND 


I .  Introduction  and  Objective 


Introduction 


In  a  1 9b i  memorandum  to  the  Secretaries  of  the  Military 
Departments,  then  Secretary  of  Defense,  Caspar  Weinberger, 
stated  that 

Too  often  in  the  past,  when  faced  with  funding  and 
schedule  constraints,  we  have  compromised  the  tech¬ 
nical  integrity  of  our  programs  by  deleting  or 
deferring  vital  program  elements  that  contribute  to 
system  performance,  producibi 1 i ty ,  and  suppor tabi 1 l ty . 

We  have  added  unintentionally  to  the  life  cycle  cost 
and  postponed  effective  operational  capability  dates 
by  pursuing  development  programs  which  do  not  yield 
producible  designs  and  supportable  configuration  in  a 
timely  manner.  (1:1) 

The  transition  a  weapon  system  makes  from  the  develop¬ 
ment  phase  to  production  is  critical  to  its  becoming  a 
successful  acquisition.  A  weapon  system  is  prepared  for 
production  during  this  transition  process.  Emphasis  is 
placed  on  finalizing  the  design  of  the  system;  establishing 
effective  test  measurements  for  the  system’s  successful  per¬ 
formance;  anc  certifying  manufacturing  planning  and  process¬ 
es  to  produce  ohe  system.  As  indicated,  this  transition  is 
a  process,  not  a  discrete,  event  whose  success  is  measured 
by  passing  a  milestone  review  at  the  end  of  the  development 
phase.  Rather,  transition  is  a  culmination  of  efforts  in 
preparing  the  system  for  production.  Ideally,  this 
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transition  process  should  start  early  in  the  full-scale 
development  (FSD)  phase  and  continue  to  full-rate  produc¬ 
tion.  The  transition  process  is  very  broad  and  is  highly 
influenced  by  the  extent  of  production  planning  early  in  the 
development  phase.  Therefore,  management  of  weapon  systems 
from  the  development  phase  through  production  requires 
effective  management  and  coordination  across  many  functions 
of  program  management. 

The  Department  of  Defense  (DOD)  policy  regarding  pro¬ 
duction  management  reflects  the  importance  of  early  produc¬ 
tion  planning  and  the  critical  nature  of  the  transition 
process.  DOD  Directive  4245.7,  "Transition  of  Weapon 
Systems  from  Development  to  Production,"  states  the  DOD  policy 
that  production  risks  be  identified  as  early  as  possible; 
that  these  risks  be  reduced  before  a  production  decision; 
that  production  engineering  and  production  planning  be  done 
throughout  FSD;  that  voids  in  production  technologies  be 
identified  and  addressed;  and  that  before  proceeding  into 
production,  contractors  demonstrate  the  capabilities  to 
produce  within  cost  and  on  schedule  (2:  1-5)  . 

One  of  the  most  important  reviews  used  by  DOD  during 
the  transition  process  is  the  Production  Readiness  Review 
(PRR) .  The  objective  of  the  PRR  is  to  determine  if  the 
weapon  system  design,  production  planning,  and  associated 
preparations  for  producing  the  system  have  progressed  to  a 
point  where  a  production  commitment  can  be  made  without 
incurring  unacceptable  risks  of  breaching  the  thresholds  of 


2 


coat,  schedule,  or  performance  (3:1).  The  PRR  consists  of  a 
team  of  experts  from  the  various  functions  of  program 
management  and  are  usually  selected  from  the  System  Program 
Office  (SPO) ,  Contract  Administration  Office  (CAO) ,  and 
other  functions  as  required. 

The  typical  PRR  will  examine  the  design  of  the  weapon 
system  for  completeness  and  produci bi 1 i ty .  More  specifical¬ 
ly,  the  team  will  review  the  contractor's  production  plan¬ 
ning,  existing  and  planned  facilities,  tooling,  manufactur¬ 
ing  methods,  material  and  manpower  resources,  production 
engineering,  and  overall  production  management  (3:3).  At 
the  completion  of  the  review,  the  PRR  team  director  Issues  a 
report  to  the  weapon  system  program  manager  which  summarizes 
potential  risks  in  each  area,  production  deficiencies,  and 
an  overall  assessment  of  transition  activities.  The  program 
manager  includes  this  evaluation  in  hi3  assessment  of  prog¬ 
ram  performance  and  risk  to  the  required  higher  level  review 
boards  as  determined  by  program  size  (4:2).  DOD  policy 
requires  that  a  PRR  be  conducted  prior  to  the  start  of 
production,  including  low  rate  initial  production  (LRIP) 
during  FSD. 

Recent  years  have  witnessed  an  increasing  emphasis  in 
upgrading  producibi 1 i ty ,  reliability,  and  maintainability  of 
new  weapon  systems  being  procured  by  DOD.  This  increased 
interest  can  be  attributed  primarily  to  the  skyrocketing 
costs  of  new  weapon  systems  and  Congressional  pressure  to 
reduce  costs.  Even  with  the  renewed  interest  in  the 
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transition  process,  many  of  the  sophisticated  weapon  systems 
being  procured  today  continue  to  experience  problems  during 
the  production  phase  that  result  in  cost  growth  and  schedule 
delays.  The  need  for  high-technology  weapons  coupled  with 
design  and  produci bi 1 i ty  problems  have  invariably  resulted 
in  DOD  receiving  systems  less  capable  than  the  original 
requirements  and  at  a  greater  cost. 

In  1982,  the  Defense  Science  Board  (DSB)  was  tasked  to 
address  similar  issues.  In  addition  to  the  escalating  cost 
of  weapon  systems  and  budget  pressures  was  the  unsatisfac¬ 
tory  effectiveness  of  producing  these  systems.  Consequent¬ 
ly,  a  DSB  Task  Force  on  'Transition  of  Weapons  Systems  from 
Development  To  Production'  was  organized  to  review, 
evaluate,  and  make  recommendations  on  ways  to  improve  and 
accelerate  the  transition  of  weapon  systems  into  production. 
The  task  force  concluded  that  the  causes  of  acquisition  risk 
in  the  transition  from  development  to  production  of  weapon 
systems  were  more  technical  in  nature  than  managerial.  In 
other  words,  reorganizing  the  milestones  of  the  acquisition 
process  was  not  the  answer.  The  Task  Force  also  concluded 
that  reducing  these  acquisition  risks  requires  the  approach 
of  designing  the  system  with  production  in  mind.  Too  often, 
weapon  systems  with  inherent  design  deficiencies  were 
allowed  to  transition  into  production  with  hopes  that 
unproven  manufacturing  technologies  would  improve  the 
system’s  produeibi 1 i ty .  The  DSB  recommendation  to  correct 
this  situation  is  a  disciplined  engineering  department 
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that  interfaces  with  manufacturing  engineering  early  in 
the  program  to  ensure  a  producible  design  before  entering 
production  (4:1-3). 

The  DSB  Task  Force  developed  a  matrix  of  the  most 
critical  events  in  the  design,  test,  and  production  elements 
of  the  production  process.  From  this  matrix  of  critical 
events,  a  set  of  templates  were  developed  which  identify 
different  levels  of  technical  risk  and  describe  techniques 
for  reducing  or  eliminating  these  risks  These  templates 
reflect  the  experience  and  knowledge  of  highly  respected 
experts  from  the  defense  industry  that  participated  in  the 
DSB  study  on  transition.  The  DSB  then  recommended  the  use 
of  these  templates  as  a  tool  to  assist  in  the  transition  of 
weapon  systems  into  production  (4:1-6). 

Justification  for  Research 

The  critical  nature  of  the  transition  of  weapons  into 
production  and  the  importance  of  the  PRR  are  further  empha¬ 
sized  by  the  large  financial  commitments  that  typically 
accompany  a  production  decision.  Approximately  one-half  of 
the  overall  defense  budget  is  accounted  for  in  the  produc¬ 
tion  phase  of  weapon  system  acquisition  (5:1).  However,  it 
is  also  during  this  phase  that  weapon  system  producibi 1 i ty 
problems  usually  arise  or  production  planning  deficiencies 
are  detected.  These  problems  usually  result  in  engineering 
design  changes  or  lower  production  yields  which,  in  turn, 
drive  production  costs  up  and  delay  delivery  schedules. 
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The  objective  of  PRRs  is  to  ensure  that  both  the  weapon 
system  design  and  contractor  are  prepared  to  enter  into 
production  with  acceptable  cost,  schedule,  and  performance 
risks  holds  (2:11-4)  .  Several  sources  suggest  that  in 
general ,  this  objective  is  not  being  met. 

First,  cost  overruns  and  schedule  delays  on  major 
weapon  systems  continue  to  fill  newspaper  headlines.  More 
recently,  the  Air  Force  B-2  bomber  and  the  Peacekeeper 
inter-continental  ballistic  missile  (MX)  programs  have  come 
under  fire  for  significant  design  problems,  test  fail¬ 
ures,  schedule  delays  and  cost  increases.  Second,  problems 
similar  to  these  were  reflected  in  the  1985  General 
Accounting  Office  (GAO)  study  entitled  'Why  Some  Weapon 
Systems  Encounter  Production  Problems  and  Others  Do  Not." 

In  their  study,  GAO  investigators  concluded  that  inadequate 
production  preparations  during  the  development  phase  led  to 
problems  in  production.  In  four  of  six  case  studies  of 
weapon  system  acquisitions,  production  preparations  were 
sporadic,  underfunded,  and  conducted  too  late  in  the 
development  phase  (6:ii).  Additionally,  PRRs  were  applied 
quite  differently  in  the  six  programs.  In  some  cases,  PRRs 
were  used  to  assess  production  readiness  and  manage  concerns 
and  risks  while  in  others  they  were  viewed  as  a  "gate  to 
pass  through"  or  a  "square  to  fill"  before  production 
(6:27).  The  programs  that  did  not  use  PRRs  as  a  tool  in 
managing  program  production  readiness  and  ri3ks  suffered 
significant  cost  and  schedule  problems.  From  their  study, 
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the  GAO  recommended  using  PRRs  as  a  tool  for  managing 
production  preparations  to  progressively  reduce  production 
risks.  In  addition,  GAO  investigators  recommended  that  PRRs 
be  conducted  at  intervals  beginning  early  in  FSD  and  they 
be  continued  throughout  preparations  for  full-rate  produc¬ 
tion  (6 : vi i )  . 

The  GAO  recommendations  on  Implementation  of  PRRs  coup¬ 
led  with  the  use  of  the  transition  templates  from  the  DSB 
study  should  have  provided  a  means  of  reducing  production 
risks,  schedule  delays,  and  cost  overruns. 

Research  Objective 

The  specific  objectives  of  this  research  are: 

1.  Review  how  PRRs  are  planned  and  implemented  during  the 
transition  from  development  to  production  of  selected  weapon 
systems  within  Air  Force  Systems  Command  (AFSC) . 

2.  Determine  if  the  recommendations  on  PRR  implementation 
and  management  from  the  1985  GAO  study  were  implemented. 

3.  Review  how  the  transition  templates  from  the  1982  DSB 
study  are  being  applied  in  the  PRR  process. 

The  following  investigative  questions  will  be  answered 
to  help  achieve  these  research  objectives. 

Investigative  Questions 

1.  How  are  PRRs  implemented  and  managed  by  the  different 
programs  among  the  product  divisions  of  AFSC? 

2.  Is  there  adequate  guidance  and  direction  provided  in  DOD 
and  Air  Force  Directives  and  Regulations  for  planning  and 
implementing  PRRs? 
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3.  What  criteria,  policies,  or  procedures  are  currently 
used  by  the  different  program  management  offices  in  assess¬ 
ing  the  readiness  of  their  weapon  system  and  contractor  to 
begin  production? 

4.  Do  these  criteria,  policies,  and  procedures  reflect  the 
recommendations  of  the  1985  GAO  study  and  1982  DSB  study 
regarding  transition  into  production? 

5.  How  are  the  transition  templates  from  DOD  Manual  4245.7, 
"Transition  From  Development  to  Production,"  applied  to  PRRs 
by  selected  program  management  offices? 

6.  What  contractual  requirements  are  placed  on  the  contrac¬ 
tor  to  plan  for  and  support  PRRs? 

7.  Of  what  perceived  value  are  PRRs  to  program  managers  in 
transitioning  their  programs  into  production? 


Scope  and  Limitations 

Although  each  of  the  branches  of  service  within  DOD 
manage  major  weapon  systems  acquisitions,  each  has  a  differ¬ 
ent  management  structure  and  program  management  offices 
spread  across  the  country.  Due  to  resource  constraints, 
this  research  will  be  limited  to  those  weapon  systems  acqui¬ 
sitions  managed  by  the  Air  Force.  However,  all  DOD  weapon 
system  acquisitions  must  adhere  to  DOD  Directives  and  Regu¬ 
lations.  Therefore,  it  is  felt  that  this  study  will  provide 
a  generalization  of  the  implementation  and  management 
practices  of  PRRs  based  on  the  applicable  DOD  policies, 
directives,  and  actual  practices. 

Summary 

Chapter  I  introduced  the  process  a  weapon  system  under¬ 
goes  as  it  transitions  from  full-scale  development  to 
production,  and  the  critical  importance  of  this  transition 
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process  in  determining  the  success  of  the  program.  In 
addition,  PRRs  were  introduced  as  the  most  visible  tool  used 
by  DOD  in  determining  the  readiness  of  a  weapon  system  and 
its  producer  to  enter  the  production  phase.  The  importance 
of  the  transition  process  and  PRRs  was  further  reinforced  by 
the  findings  from  the  1982  DSB  and  1985  GAO  studies  regard¬ 
ing  transition.  Next,  the  Justification  for  the  research 
was  reinforced  with  evidence  of  programs  encountering  pro¬ 
duction  problems  stemming  from  failure  to  use  PRRs  as  a 
management  tool  during  development.  Also,  the  research 
objective  of  determining  how  PRRs  are  planned  and  implement¬ 
ed  among  the  different  programs  within  AFSC  was  introduced. 

A  review  of  whether  the  recommendations  were  implemented 
from  both  the  1982  DSB  study  and  the  1985  GAO  study  is 
included  in  the  research  objective.  Finally,  the  scope  and 
limitations  of  this  research  were  delineated. 
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Literature  Review  and  Research  Methodology 


II  . 


Overview 

This  chapter  describes  the  DOD  acquisition  cycle  with 
its  five  phases  and  corresponding  milestones.  Next,  the 
transition  of  weapon  systems  from  development  to  production 
is  discussed  including  when  and  where  it  occurs  during  the 
acquisition  cycle.  The  PRR  is  then  described  in  detail 
including  its  importance  as  a  tool  for  DOD  to  determine 
production  readiness  during  transition.  Two  DOD  studies 
concerning  transition  risks  and  production  problems  are 
discussed  including  conclusions  and  recommendations  to 
preclude  future  problems.  Following  this  discussion  is  a 
description  of  AFSC  and  its  six  product  divisions  with  its 
mission  of  developing,  acquiring,  and  delivering  weapon 
systems  the  Air  Force  needs  to  meet  its  mission.  Finally, 
the  methodology  used  to  conduct  this  research  is  discussed 
to  close  out  the  chapter. 

Literature  Review 

Weapon  System  Acquisition.  Weapon  system  acquisition 
is  a  complex  and  critical  process  in  the  DOD  with  signifi¬ 
cant  impacts  on  both  national  defense  and  the  national  budget. 
The  efficiency  and  effectiveness  of  this  process  determines 
how  well  U.S.  taxpayers’  money  is  spent  or,  in  the 
vernacular,  how  much  "bang  for  the  buck"  is  achieved. 
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DOD  policy  for  major  weapon  system  acquisition  is 
outlined  in  DOD  Directive  5000.1,  ‘Major  and  Non-Major 
Defense  Acquisition  Programs."  The  directive  states  in 
part : 

The  policy  of  the  Department  of  Defense  is  to 
assure  that  the  DOD  Acquisition  System  functions 
in  a  timely,  efficient  and  effective  manner  to 
achieve  the  operational  objectives  of  U.S.  Armed 
Forces  in  support  of  national  policies  and 
ob j  ect i ves .  (7:3) 

Implementation  and  management  of  the  weapon  system 
acquisition  process  normally  occurs  in  five  phases.  These 
phases  enhance  management  effectiveness  and  can  be  tailored 
for  each  system  to  minimize  acquisition  time  and  life  cycle 
costs.  The  tailoring  of  these  phases  are  primarily  depen¬ 
dent  on  urgency  of  need,  degree  of  technical  risk,  and 
demonstrated  progress  of  the  weapon  system  (8:2).  The  end 
of  each  phase  of  acquisition  coincides  with  a  milestone 
decision  to  either  enter  the  next  phase  of  acquisition  or 
end  the  program.  This  milestone  decision  for  major  DOD 
programs  is  ultimately  made  by  the  Secretary  of  Defense 
assisted  by  recommendations  and  advisement  of  the  Defense 
Systems  Acquisition  Review  Council  (DSARC) (7 : 4) .  For  major 
acquisitions  within  the  Air  Force  and  as  directed  by  the 
Secretary  of  Defense,  the  milestone  decisions  are  made  by 
the  Secretary  of  the  Air  Force  assisted  by  the  recommenda¬ 
tion.-;  . md  advisement  of  the  Air  Force  Acquisition  Review 
Council  (AFSARC) .  The  DSARC  and  AFSARC  are  the  top  level 
DOD  and  Air  Force  corporate  bodies  for  weapon  system 
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acquisition  (9:2).  The  decision  on  whether  to  enter  the 
next  phase  of  acquisition  is  based  on  thresholds  of  cost, 
schedule,  and  performance. 

Figure  1  depicts  the  major  milestones  and  phases  of 
the  weapon  system  acquisition  process.  The  acquisition 
process  is  initiated  with  a  mission  analysis.  This  analysis 
evaluates  operational  needs  and  the  evolving  technology 
base.  At  milestone  0,  the  decision  is  made  on  whether  a 
mission  need  exists  that  warrants  further  study.  If  this 
need  does  exist,  the  decision  is  made  to  enter  the  Concept 
Exploration/Definition  phase. 

During  Concept  Exploration/Definition,  a  commitment  is 
made  to  identify  and  explore  alternative  solutions  to  the 
needs  identified  in  the  mission  analysis.  The  main  charac¬ 
teristic  of  this  phase  is  the  solicitation  for  proposed 
solutions  according  to  mission  needs  and  not  system  charac¬ 
teristics.  Therefore,  establishment  of  a  system  acquisition 
program  during  this  phase  does  not  necessarily  mean  a  new 
system  will  be  acquired.  At  the  completion  of  this  phase, 
the  Milestone  I  decision  is  made  on  whether  to  enter  the 
Demonstration  and  Validation  phase.  This  decision  is  based 
on  the  following  considerations:  program  trade-offs;  perfor¬ 
mance,  cost,  and  schedule  trade-offs;  and  affordability  and 
life-cycle  costs  (9:3). 

During  Demonstration  and  Validation,  the  practicality 
and  value  of  the  alternative  solution  chosen  during  the 
previous  phase  is  expanded  and  studied. 
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Figure  1.  Weapon  System  Acquisition  Cycle  (8:3) 


This  study  usually  occurs  through  system  prototyping,  paper 
studies,  and/or  subsystem  prototyping.  The  Milestone  II 
decision  occurs  at  the  end  of  this  phase  and  determines 
whether  to  enter  the  Full-Scale  Development  (FSD)  phase,  and 
when  appropriate,  low  rate  initial  production  (LRIP) . 

During  FSD,  the  primary  focus  is  on  baselining  the 
design  of  the  weapon  system  and  preparing  the  system  and 
its  producer  to  begin  production.  Included  in  this  phase 
is  the  design  and  development  of  all  required  support  equip¬ 
ment  and  documentation  needed  to  produce  and  field  the 
system.  The  Milestone  III  decision  occurs  at  the  end  of 
this  phase  and  determines  whether  to  enter  the  Full-Rate 
Production  and  System  Deployment  phase  (9:4). 

During  Full-Rate  Production  and  System  Deployment,  the 
weapon  system,  training  equipment,  spares,  and  other  support 
equipment  are  produced  for  operational  use.  Also  during 
this  phase,  program  management  respons ibi 1 i ty  transfer 
(PMRT)  occurs  to  coincide  with  deployment  of  the  system 
and  allow  for  field  support.  The  Milestone  IV  decision 
occurs  approximately  1-2  years  after  initial  deployment  to 
determine  whether  operational  readiness  and  support  objec¬ 
tives  are  achieved  after  several  years  of  operational 
service  (9:4) . 

Approximately  5-10  years  after  initial  operational 
capability  is  achieved,  the  Operational  Support  phase 
occurs.  The  purpose  of  this  phase  is  to  determine  whether 
the  fielded  system  warrants  a  major  upgrade  or  replacement. 
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The  primary  considerations  during  this  phase  are  the  capa¬ 
bility  of  the  system  to  continue  to  meet  mission  require¬ 
ments,  changes  in  threat,  and  assess  new  technology  that 
offers  significant  increases  in  capability  (9:4). 

Transition  From  Development  To  Production 

Transition  of  a  weapon  system  from  FSD  into  full-rate 
production  is  a  broad  and  complex  process.  During  this 
process,  a  weapon  system  is  prepared  for  production. 
Emphasis  is  placed  on  finalizing  the  design  of  the  system; 
establishing  effective  test  measurements  for  the  system’s 
successful  performance;  and  certifying  manufacturing 
planning  and  processes  to  produce  the  system.  The  transi¬ 
tion  process  is  highly  influenced  by  the  extent  of  produc¬ 
tion  planning  early  in  the  development  phase. 

Due  to  the  broad  and  complex  nature  of  the  transition 
process,  management  of  weapon  systems  from  development 
through  the  production  phase  requires  effective  management 
and  coordination  across  many  functions  of  the  program 

The  DOD  policy  regarding  transition  is  stated  in  DOD 
Directive  4245.7,  ‘Transition  From  Development  To 
Production . 

1.  Acquisition  programs  shall  be  subjected  to  a 

rigorous,  disciplined  application  of  fundamental 
engineering  principles,  methods,  and  techniques 
in  accordance  with  DOD  4245. 7-M.  Elements  of 
program  risk  shill  be  identified  and  assessed 
throughout  the  acquisition  cycle.  The  program's 
acquisition  strategy  shall  feature  provisions  for 
eliminating  or  reducing  these  risks  to  acceptable 
levels .  (2:2) 
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Transition  Problems 


Weapon  system  problems  with  performance,  quality,  and 
producibi 1 ity  frequently  do  not  surface  until  the  program 
transitions  from  FSD  to  production.  Although  the  system  may 
pass  all  of  its  milestone  reviews,  the  weapon  system  still 
may  not  succeed  due  to  technical  reasons.  These  technical 
problems  can  be  attributed  to  poor  engineering  designs. 

Poor  designs  result  in  higher  rates  of  test  failures  and 
manufacturing  problems  that  overwhelm  production  schedules 
and  costs.  These  problems  lead  to  the  “hidden  factory' 
syndrome  of  high  redesign  and  rework  rates.  Also,  field 
failures  could  result  and  hamper  operational  and  training 
schedules . 

The  main  contributor  to  transition  risk  and  production 
problems  is  the  emphasis  on  technical  performance  of  a 
system  during  development  at  the  expense  of  producibi 1 1 ty , 
reliability,  and  maintainability.  Since  technical 
performance  is  the  yardstick  for  success  during  FSD,  the 
tendency  is  to  overdesign  to  ensure  adequate  technical 
performance  is  met  and  ensure  success  during  this  phase. 
Consequently  production  planning  is  given  a  lesser  priority 
than  required  during  FSD. 

Some  of  the  major  factors  attributed  to  lack  of  early 
production  planning  in  FSD  and  ultimately  affect  a 
successful  transition  are  as  follows: 

1.  Des ign  Changes ■  Due  to  design  refinements  and 
correction  of  design  deficiencies,  the  weapon  system 
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baseline  configuration  is  continuously  changing.  These 
design  changes  can  be  expensive  to  incorporate  and  cause 
scrapping  and  reordering  of  components  and  raw  materials. 

In  addition,  design  changes  often  require  redesign  of 
production  processes  and  tooling  (6:1). 

2.  Long  Leadtimes.  Material  leadtimes  can  be  affected 
by  both  design  changes  and  poor  vendor  performance.  Design 
changes  often  require  ordering  alternative  parts  and  com¬ 
ponents  to  support  these  changes.  Certain  items  such  as 
forgings,  precision  castings,  special  alloys  and  electronic 
components  have  leadtimes  ranging  from  6  months  up  to  2 
years  (10:7).  Therefore,  design  changes  requiring  material 
changes  could  have  a  significant  impact  on  production  costs 
and  schedules.  Poor  vendor  performance  is  a  problem  which 
is  less  manageable  and  is  compounded  by  optimistic  materials 
schedules  that  worsen  the  effect  of  late  deliveries  (10:7-8) . 

3.  Poor  Quality.  Higher  rates  of  hardware  and  test 
failures  can  be  attributed  to  continuous  design  changes  and 
lack  of  emphasis  on  produc i b i 1 i ty  with  respect  to  these 
changes  (10:7-8) . 

4.  Hidden  Factory  Syndrome.  A  result  of  any  or  all  of 
the  previously  mentioned  problems  contributes  to  a  large 
portion  of  the  production  facility,  tooling,  and  personnel 
being  consumed  repairing,  reworking,  and  retesting 
discrepant  hardware  or  test  failures.  The  net  effect  is  a 
decrease  in  production  capacity  and  adverse  impacts  on  pro¬ 
duction  costs  and  schedules  (10:8). 
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The  transition  from  development  to  production  of  new 
weapon  systems  is  an  important  and  complex  process  which 
should  begin  early  in  FSD  and  continue  through  production. 
Integrating  and  incorporating  the  design,  test,  and 
production  planning  early  in  FSD  is  vital  to  a  smooth  and 
successful  transition.  To  support  a  successful  transition, 
an  adequately  planned  and  funded  acquisition  plan  which 
includes  managing  transition  risks  is  essential. 

Production  Readiness  Reviews 

One  of  the  most  important  reviews  used  by  DOD  to 
identify  and  manage  production  risks  during  transition  is 
the  PRR.  The  PRR  concept  evolved  over  a  number  of  years  in 
response  to  the  growing  need  of  determining  whether  a  weapon 
system  design  and  its  contractor  were  ready  to  begin 
production.  Beginning  in  1977,  DOD  Directive  5000.34, 
"Defense  Production  Management,"  focused  attention  on  mini¬ 
mizing  transition  risks  by  specifically  assigning  production 
management  responsibilities.  These  responsibilities  in¬ 
cluded  ensuring  that  at  the  end  of  FSD,  all  factors  relevant 
to  production  were  considered  and  that  the  weapon  system  was 
ready  for  production.  The  basic  criteria  for  entering  pro¬ 
duction  as  outlined  in  this  directive,  was  minimizing  risks 
of  breaching  cost,  schedule  and  performance  thresholds.  In 
1979,  DOD  Instruction  5000.38,  "Production  Readiness 
Reviews,"  followed  which  further  outlined  the  purpose, 
scope,  and  procedures  for  conducting  PRRs . 
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DOD  policy  governing  the  implementation  and  management 

/ 

of  PRRs  requires  that  a  PRR  be  conducted  before  production 
begins,  including  low  rate  initial  production  during  FSD . 

The  findings  of  the  PRR  will  be  provided  to  the  highest 
level  review  body,  either  the  DSARC  or  AFSARC  as  required 
to  support  the  production  decision  (3:1). 

PRR  Objective 

The  purpose  of  the  PRR  is  to  determine  if  the  weapon 
system  design,  production  planning,  and  associated  prepara¬ 
tions  for  producing  the  system  have  progressed  to  a  point 
where  a  production  commitment  can  be  made  without  incurring 
unacceptable  cost,  schedule  or  performance  risks  (3:1). 
Production  readiness  occurs  when  all  engineering  and 
manufacturing  problems  have  been  resolved  or  managed,  and 
production  planning  has  been  reviewed  in  sufficient  detail 
to  minimize  production  risks  (8:6). 

PRR  Process 

A  typical  PRR  examines  the  weapon  system  design  for 
completeness  and  produc ibi 1 i ty .  More  specifically,  the  PRR 
evaluates  the  contractor's  production  planning,  existing  and 
planned  facilities,  tooling,  manufacturing  methods,  material 
and  manpower  resources,  production  engineering,  and  overall 
production  management  (3:3).  The  PRR  evaluation  process 
consists  of  hands-on  examinations  of  these  functions  and 
typically  follows  the  flow  illustrated  in  Figure  2. 
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Also,  included  in  this  review  were  other  functions  which  could 
have  an  effect  on  production  costs  and  schedules. 

The  evaluation  criteria  or  specific  objectives  varies 
between  PRRs  and  is  based  on  the  complexity  of  the  weapon 
system  being  produced;  technical  progress  of  the  program; 
and  the  PRR  team  make-up.  Once  each  function  or  element 
identified  for  review  is  evaluated,  a  determination  is  made 
as  to  the  production  risk  of  each  element.  Then  a  cumula¬ 
tive  production  risk  is  determined  from  all  the  reviewed 
functions  combined.  At  the  completion  of  the  review,  the 
PRR  team  director  issues  a  report  to  the  weapon  system 
program  manager  which  summarizes  any  risk  potential ,  produc¬ 
tion  deficiencies,  and  an  overall  assessment  of  transition 
activities.  If  the  PRRs  are  conducted  incrementally,  these 
risks  and  concerns  can  be  reduced  or  managed  prior  to  the 
production  decision.  The  program  manager  includes  the  final 
evaluation  in  his  assessment  of  technical  performance  and 
transition  risks  to  the  required  higher  level  review  boards 
to  support  the  production  decision  recommendation  (3:1). 

The  initial  PRR  of  a  new  weapon  system  should  be 
conducted  within  90  days  of  the  Critical  Design  Review 
(CDR)  and  the  final  PRR  is  required  at  least  120  days  prior 
to  the  Milestone  III  production  decision  (8:99). 

When  several  PRRs  are  conducted,  they  should  be  incremental 
with  an  interval  of  at  least  12  months.  The  number  of  PRRs 
to  be  conducted;  the  specific  objectives  of  each  PRR;  and 
the  time  phasing  of  the  PRRs  is  based  on  the  length  and 
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complexity  of  the  program  along  with  the  performance  of  the 
contractor . 

PRR  teams  consist  of  experts  from  the  various  func¬ 
tions  of  manufacturing  management  and  program  management. 
These  experts  are  typically  chosen  from  the  System  Program 
Office  (SPO) ,  Contract  Administration  Office  (CAO) ,  weapon 
system  contractor,  and  other  personnel  as  required  (3:3). 

The  PRR  team  is  usually  divided  into  panels  based  on  func¬ 
tional  areas  to  be  reviewed  (See  Figure  3) .  These  panels 
consist  of  personnel  who  review  specific  functional  areas 
during  the  review.  The  number  of  panels  varies  between  PRRs 
and  appears  to  be  dependent  on  the  same  criteria  as  the  team 
size  . 

Obtaining  contractor  participation  requires  contrac¬ 
tual  coverage.  This  coverage  is  usually  specified  in  the 
Statement  of  Work  (SOW)  in  the  FSD  contract  (11:11-17). 
Included  in  these  requirements  should  be  the  number  of  PRRs 
required;  extent  of  government  and  contractor  participation; 
responsibility  for  subcontractors  PRRs;  and  overall  direc¬ 
tion  and  purpose  of  the  PRRs. 

Defense  Science  Board  Task  Force  Study 

DSB  Task  Force  Objective.  Although  PRRs  were  incor¬ 
porated  in  the  transition  process,  new  weapon  systems  con¬ 
tinue  to  encounter  difficulties  when  beginning  production. 
These  continuing  transition  problems  coupled  with  a  rapidly 
increasing  interest  in  upgrading  the  reliability  and 
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maintainability  of  new  weapon  systems  called  for  a  review  of 
the  acquisition  cycle.  In  response  to  these  concerns,  the 
Defense  Science  Board  (DSB)  Task  Force  on  Transition  of 
Weapon  Systems  From  Development  to  Production  was  establish¬ 
ed  in  June  of  1982.  The  purpose  of  the  DSB  Task  Force  was 
threefold.  First,  examine  ways  and  methods  to  accelerate 
and  improve  the  transition  process;  second,  study  both 
government  and  industry  management  of  the  transition  pro¬ 
cess;  and  third,  make  recommendations  relating  to  design, 
test,  and  production  that  will  result  in  the  timely  delivery 
of  quality  products  to  the  government  (12:1).  Included  in 
this  study  was  an  evaluation  of  not  only  the  technical 
aspects  of  the  weapon  system  design  and  manufacturing 
processes ,  but  also  the  funding  and  managerial  aspects  of 
the  program  (12: ix). 

DSB  Task  Force  Approach.  The  DSB  Task  Force  consisted 
of  top  level  experts  from  engineering,  production,  and  over¬ 
all  program  management  from  both  government  and  industry. 
These  individuals  possessed  extensive  experience  with  DOD 
acquisitions  and  the  transition  process.  The  approach  taken 
by  the  Task  Force  was  the  formation  of  five  panels  to 
evaluate  and  assess  issues  related  to  design,  test,  produc¬ 
tion,  facilities,  and  management  of  the  weapon  system.  Each 
panel  was  assigned  specific  issues  and  areas  for  detailed 
analysis.  The  focus  of  the  study  initially  evolved  around 
the  fundamental  principles  and  guidelines  established 
through  DOD  policies  and  directives.  However,  after  the 


24 


study  started,  the  Task  Force  determined  that  some  DOD 
policies  and  directives  required  change. 

Upon  completion  of  the  study,  each  panel  submitted  a 
report  of  conclusions  and  recommendations  to  the  Task  Force 
Chairman  who  in  turn,  integrated  and  coordinated  the  conclu¬ 
sions  and  recommendations  into  the  Task  Force  Report. 

DSB  Task  Force  Conclusions  and  Recommendations.  The 
DSB  Task  Force  concluded  that  the  major  causes  of  acquisi¬ 
tion  risk  in  the  transition  from  development  to  production 
of  weapon  systems  were  more  technical  in  nature  than 
managerial.  In  addition,  the  Task  Force  concluded  that 
these  risks  required  the  approach  of  designing  the  system 
with  production  in  mind  (4:1-3).  Too  often,  weapon  systems 
with  inherent  design  deficiencies  were  allowed  to  transition 
into  production  with  hopes  that  unproven  manauf actunng 
technologies  would  improve  the  system’s  producibi 1 1 ty .  The 
DSB  recommended  approach  to  correct  this  situation  is  a 
disciplined  engineering  department  that  is  interrelated  with 
manufacturing  engineering  to  factor  producibi 1 i ty  into  the 
design  early  on  in  FSD  (12:3). 

Included  in  the  DSB  Task  Force  report  were  a  set  of 
templates  ,  or  checklists,  that  describe  techniques  for 
improving  the  acquisit-on  process.  The  key  to  these 
templates  is  the  fact  that  they  treat  the  acquisition 
process  as  an  industrial  process  involved  with  the  design, 
test,  and  production  of  low  risk  weapon  systems.  Each 
template  describes  an  area  of  risk  and  also  managerial 
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methods  for  reducing  or  managing  that  risk.  The  templates 
were  developed  from  lessons  learned,  analysis  of  a  number  of 
programs,  and  the  technical  expertise  of  the  Task  Force 
members  ( 4 : 1 -6 ) . 

General  Accounting  Office  Study 

GAO  Study  Objective.  Continued  problems  of  cost 
growth  and  late  deliveries  on  new  weapon  systems  entering 
production  prompted  a  1985  General  Accounting  Office  (GAO) 
study  entitled,  “Why  Some  Weapon  Systems  Encounter 
Production  Problems  While  Others  Do  Not:  Six  Case  Studies." 

The  objective  of  the  GAO  study  was  to  examine  six 
weapon  system  acquisition  programs  and  determine  the  causes 
of  early  production  problems  and  outline  corrective  actions 
to  preclude  their  occurence  in  future  programs  (6:i).  In 
the  GAO  study,  investigators  concluded  that  inadequate 
production  preparations  during  the  development  phase  led  to 
problems  in  production.  In  four  of  the  six  programs 
Studied,  production  preparations  were  sporadic,  underfunded, 
and  conducted  too  late  in  FSD  (6:ii).  Additionally,  PRRs 
were  applied  were  quite  differently  in  the  six  programs.  In 
some  cases,  PRRs  were  used  to  assess  production  readiness 
and  manage  concerns  and  risks  while  in  others  they  were  used 
as  a  'gate  to  pass  through'  or  'square  to  fill"  before 
production  (6:27),  The  programs  that  did  not  use  PRRs  as  a 
tool  for  assessing  and  managing  production  readiness 
Suffered  significant  cost  and  schedule  problems. 
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GAO  Study  Recommendations.  Based  on  the  findings  from 
their  study,  GAO  investigators  made  the  following  recommen¬ 
dations  to  reduce  the  occurrence  of  problems  early  in  the 
production  phase. 

1.  Ensure  that  an  adequate  level  of  production 
preparations  are  made  early  and  continuously  through  FSD. 

2.  Ensure  that  when  funding  or  requirements  changes 
affecting  production  are  made  in  a  program  that  the  risks 
are  assessed  before  making  these  changes. 

3.  Use  PRRs  as  a  tool  in  assessing  and  managing 
production  readiness  and  risks.  PRRs  should  begin  early  in 
FSD  and  be  repeated  at  regular  intervals  until  production 
begins  (6 : vi i ) . 

Air  Force  Systems  Command 

Mission ■  Air  Force  Systems  Command  (AFSC) ,  headquar¬ 
tered  at  Andrews  AFB ,  Md . ,  is  responsible  for  developing, 
acquiring,  and  delivering  weapon  systems  the  Air  Force  needs 
to  meet  its  mission  requirements.  The  command’s  emphasis 
is  on  delivering  superior  systems  that  are  reliable,  main¬ 
tainable,  and  supportable.  Through  its  six  product 
divisions,  AFSC  strives  to  get  capable  weapon  systems  off 
the  drawing  board  and  into  the  field  as  quickly  and  cost 
effectively  as  possible.  In  addition,  AFSC  also  handles  the 
major  aerospace  responsibilities  of  DOD  including  research 
and  development  of  satellites,  boosters,  space  probes,  and 
associated  systems  for  NASA  (13:1). 
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AFSC  Product  Divisions 


Aeronautical  Systems  Division  (ASP) .  Located  at 
Wr ight-Patterson  AFB ,  OH.,  ASD  is  the  largest  product 
division  within  AFSC.  ASD  is  responsible  for  developing  and 
acquiring  aeronautical  systems  and  related  equipment  for  the 
Air  Force.  With  a  1987  budget  of  *16.8  billion  and  a  staff 
of  more  than  11,000  employees,  some  of  the  major  programs 
under  ASDs  control  include:  the  Bl-B  Strategic  Bomber;  B-2 
Advanced  Technology  Bomber;  Advanced  Tactical  Fighter;  and 
the  C-17  Airlifter.  ASD’s  priorities  include  strengthening 
nuclear  and  conventional  forces,  expanding  airlift  capabili¬ 
ties,  and  modernizing  the  tactical  air  forces  (14:1). 

Munition  Systems  Division  (MSP).  Located  at  Eglin 
AFB,  Fla.,  MSD  is  responsible  for  planning,  researching, 
developing,  and  acquiring  air  armaments  and  related  equip¬ 
ment.  Included  in  MSD ’ s  mission  area  is  a  test  wing  equip¬ 
ped  with  test  aircraft  allowing  MSD  to  manage  new  air  arma¬ 
ments  from  'cradle  to  grave.'  Some  major  programs  currently 
under  MSD's  control  include:  the  Advanced  Medium  Range  Air 
to  Air  Missile  (AMRAAM) ;  and  the  Sensor  Fuzed  Weapon ( 1 5 : 1 ) . 

Electronic  Systems  Division  (ESP).  Located  at  Hanscom 
AFB,  MA.,  ESD  is  responsible  for  developing  and  acquiring 
command,  control  and  communications  systems  for  the  Air 
Force.  With  an  annual  budget  of  over  *4  billion,  ESD  works 
with  operational  forces  to  harness  electronic  technology 
for  ensuring  effective  use  of  military  personnel  and 
weaponry  (16:1).  Some  of  the  major  programs  currently  under 
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the  control  of  ESD  include:  the  Airborne  Warning  and  Control 
System  (AWACS) ;  and  modernization  of  the  North  American 
Aerospace  Defense  Command  Center. 

Ballistic  Systems  Division  (BSD).  Located  at  Norton 
AFB ,  CA. ,  BSD  is  responsible  for  developing  and  acquiring 
intercontinental  ballistic  missiles  (ICBM).  With  a  work¬ 
force  of  950  military  and  civilian  employees,  BSD  integrates 
the  the  activities  of  a  number  of  major  contractors  who 
develope  and  build  portions  of  missile  systems.  Major 
programs  currently  under  the  control  of  BSD  include:  the 
Peacekeeper  Missile  in  Minuteman  Silo;  the  Peacekeeper 
Missile  in  Rai lgarrison ;  and  the  Small  ICBM  (17:1). 

Space  Systems  Division  (SSD) .  Located  at  Los  Angeles 
AFB,  CA. ,  SSD  is  responsible  for  research,  development, 
acquisition,  launch,  and  on-orbit  command  and  control  of 
military  space  systems.  In  addition,  SSD  is  the  focal  point 
within  DOD  for  plans  and  activities  associated  with  the 
military  use  of  the  Space  Shuttle.  Some  major  programs 
currently  under  the  control  of  SSD  include:  the  Defense 
Satellite  Communications  System;  the  Navstar  Global 
Positioning  Satellite;  and  the  Defense  Meteorological 
Satellite  Program  (18:1). 

Human  Systems  Division  (HSD) .  Located  at  Brooks  AFB, 
Tex. ,  HSD  is  responsible  for  ensuring  Air  Force  systems  and 
operations  are  designed  with  human  capabilities  in  mind. 

HSD  focuses  on  integrating  human  operators  into  complex 
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systems,  protecting  crews  in  hazardous  situations,  and 
preserving  the  human  environment  (13:2). 

These  six  major  product  divisions  of  AFSC  comprise  the 
major  thrust  of  new  weapon  system  acquisition  for  the  Air 
Force.  However,  HSD  was  not  considered  for  this  research 
due  its  unique  mission  of  research  and  development  of  human 
systems  capabilities  and  low  volume  production  efforts. 

Research  Methodology 

To  enable  an  in-depth  and  focused  examination  of 
actual  PRR  implementation  and  management  practices,  a  case 
study  analysis  of  ten  current  weapon  system  acquisition 
programs  was  conducted.  The  programs  chosen  provided  the 
framework  for-  hho  evaluation  of  programs  ranging  in  size 
from  relatively  small  to  major  DOD  acquisitions.  The 
programs  chosen  are  listed  in  the  following  table. 
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Table  I.  Programs  Selected  for  this  Research 


Weapon 

System 


Product 

Division 


Acq  . 

Cost 

(*  in  millions) 


QF-106  Full-Scale  MSD 

Aerial  Target 

Improved  Data  Link  MSD 

Aircraft/Weapon  Data 

Terminals 

Sensor  Fuzed  Weapon  MSD 

Air  Launched  Missile 

Advanced  Medium  Range  MSD 

Air  to  Air  Missile  (AMRAAM) 

F100PW220/F1 10GE100  ASD 

Jet  Engines 

Tacit  Rainbow  Missile  ASD 

Advanced  Cruise  Missile  ASD 

Low  Altitude  Navigation  and  ASD 
Targeting  Infrared  at  Night 
Pods  (LANTIRN) 

C-17A  Transport  Aircraft  ASD 

B-2  Advanced  Technology  ASD 

Bomber 


79 


42 

(FSD) 


22 

(FSD) 

1 ,000 
plus 
options 

300 

plus 

options 

classi f ied 

class i f led 

2,800 


36,000 

65,000 


Case  Study  Justification 

The  case  study  approach  was  chosen  to  attempt  to 
resolve  the  research  objective  for  this  study.  In  this 
particular  study,  an  attempt  was  made  to  determine  how  PRRs 
were  actually  implemented  and  managed  on  selected  programs 
within  AFSC .  Because  questions  were  posed  with  the  focus 
on  actual  PRR  applications  rather  than  D0D  regulations  and 
directives;  the  case  study  was  the  preferred  approach. 
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The  results  from  this  case  study  are  generalizations  based 
on  the  phenomenon  studied  and  do  not  represent  any  specific 
sample  or  population. 

The  goal  of  this  study  was  to  expand  and  generalize 
the  theories  of  PER  implementation  and  management  which  are 
actually  used.  The  strength  of  this  case  study  was  its 
ability  to  deal  with  a  wide  variety  of  evidence  which 
included  PER  plans,  PRR  reports,  PRR  inbrief s/outbriefs ,  and 
informal  interviews. 

Case  Study  Design 

The  case  study  designed  for  this  review  consists  of 
two  main  review  elements  (See  Appendix  A) .  These  elements 
are  documentation  review  and  short  personal  interviews. 
Questions  were  generated  which  supported  both  a  logical 
sequence  of  documentation  review  and  provided  the  data  to 
resolve  the  research  objective.  For  the  personal  interview 
element,  questions  were  generated  that  helped  determine  the 
meaningfulness  or  value  to  managers  of  the  PRRs  conducted  on 
their  programs.  In  addition,  any  recommendations  for 
improvement  of  the  PRR  process  was  sought  along  with  con¬ 
structive  criticisms.  Although  the  interview  element  was 
highly  opinionated,  it  provided  actual  managerial  insight 
concerning  PRR  management. 

Case  Study  Approach 

In  order  to  successfully  answer  the  research  objec¬ 
tive,  several  different  elements  of  actual  PRR  planning, 
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implementation,  and  management  were  reviewed.  The  following 
documents  were  reviewed  as  part  of  the  documentation  element 
of  the  research. 

Contract  Statement  of  Work  (SOW).  The  SOW  was 
reviewed  for  each  program  to  determine  what  PRR  requirements 
and  planning  were  identified  up  front  in  the  contract.  In 
addition,  the  SOW  was  reviewed  for  specific  government  and 
contractor  requirements  for  planning  and  preparations. 

PRR  Plan.  The  PRR  plan  for  each  of  the  ten  programs 
was  reviewed  for  actual  pre-PRR  planning.  These  plans  re¬ 
flected  the  number  of  PRRs  planned  for  the  program;  time 
phasing  and  team  sizes  of  PRRs;  evaluation  criteria;  and 
government  and  contractor  responsibilities. 

PRR  Inbrief s/Outbriefs ■  When  available,  the  PRR 
inbriefs  and  outbriefs  were  reviewed  for  specific  PRR 
objectives;  team  staffing  and  panel  structure;  evaluation 
criteria;  and  overall  direction  of  specific  PRRs. 

PRR  Reports .  Specific  PRR  reports  were  reviewed  for 
actual  evaluation  criteria;  identification  and  management  of 
any  transition  risks  or  production  deficiencies;  and 
overall  managerial  risk  assessment. 

Personal  Interviews.  Informal  interviews  were  con¬ 
ducted  with  the  manufacturing  managers  for  all  ten  programs 
and  with  several  program  managers.  The  interviews  provided 
the  following  management  insights:  philosophies  concerning 
PRRs;  value  of  PRRs;  implementation  and  management  strate¬ 
gies;  and  recommendations  for  improvement  or  change. 


33 


The  purpose  of  the  interviews  was  to  supplement  the 
findings  from  the  documentation  review.  Any  ambiguities  or 
deviations  from  plans  in  implementation  and  management  of 
PRRs  were  addressed  through  the  interviews. 

Scope  of  the  Research 

Due  to  resource  constraints,  the  scope  of  this  re¬ 
search  was  limited  to  selected  programs  within  AFSC .  More 
specifically,  the  programs  were  chosen  from  ASD  at  Wright- 
Patterson  AFB ,  OH.,  and  from  MSD  at  Eglin  AFB ,  FLA.  These 
two  product  divisions  were  chosen  for  several  reasons. 

First,  they  represent  the  two  largest  product  divisions 
within  AFSC  and  provided  a  greater  database  for  gathering 
the  required  PRR  data  for  analysis.  Second,  these  two 
product  divisions  are  more  oriented  towards  production  in 
terms  of  volume  and  funding.  The  remaining  product  divi¬ 
sions  deal  primarily  in  high  cost,  low  volume  products  that 
would  have  narrowed  the  research. 

The  number  of  programs  reviewed  was  also  limited  due 
to  resource  constraints.  The  ten  programs  were  chosen  based 
program  status  and  availability  of  PRR  data  that  provided 
the  best  basis  for  generalizations  of  PRR  practices.  In 
addition,  the  selected  programs  represent  a  continuum 
ranging  from  low  cost  and  high  volume  to  high  cost  and  high 
volume  weapon  systems. 
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Chapter  Summary 

The  transition  from  development  to  production  of  a 
weapon  system  is  critical  to  its  becoming  a  successful 
acquisition.  During  this  transition,  weapon  system  designs 
are  prepared  for  production.  One  of  the  most  important 
reviews  used  by  DOD  during  the  transition  process  is  the 
PRR.  However,  even  with  the  use  of  PRRs ,  new  weapon  systems 
continue  to  experience  cost  and  schedule  problems  early  in 
production.  This  was  reflected  in  both  the  1982  DSB  Task 
Force  study  and  the  1985  GAO  study  concerning  transition  and 
production  problems.  Both  of  these  studies  resulted  in 
recommendations  for  improvement  of  the  transition  process 
and  more  specific  use  of  PRRs. 

The  case  study  for  this  research  was  designed  to 
identify  the  implementation  and  management  practices  of 
selected  programs  within  AFSC.  Included  in  this  research 
was  the  determination  if  the  recommendations  from  both 
government  studies  were  being  implemented. 
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Ill . 


Research  Findings 


This  chapter  presents  the  findings  from  the  case  study 
analysis  described  in  the  previous  chapter.  Data  obtained 
from  the  informal  interviews  and  documentation  reviews  were 
analyzed  to  provide  the  basis  for  the  findings  presented. 

The  data  and  findings  are  presented  in  the  form  of  responses 
along  with  specific  details  to  the  seven  investigative 
questions  for  this  research.  In  addition,  several  questions 
required  comparison  among  product  divisions,  weapon  system 
programs,  and  individual  management  approaches.  These  com¬ 
parisons  and  generalizations  are  included  in  the  responses 
to  the  applicable  investigative  questions. 

Findings 

Investigative  Question  I.  How  are  PRRs  implemented 
and  managed  by  the  different  weapon  system  programs  among 
the  product  divisions  of  AFSC? 

A  series  of  questions  were  generated  for  the  case 
study  to  identify  and  evaluate  specific  facts  that  respond 
to  the  investigative  questions.  More  specifically,  the 
following  areas  were  addressed  to  provide  this  information: 
number  of  PRRs  planned  and  implemented;  duration  of  these 
PRRs;  PRR  team  size  and  panel  structure;  evaluation  criteria 
and  risk  assessment  categories;  risk  management;  and  general 
emphasis  or  objective  of  these  PRRs. 

First,  the  number  of  PRRs  that  were  planned  and  imple¬ 
mented  varied  among  the  ten  programs.  The  average  number  of 
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PRRs  planned  per  program  was  three  with  an  average  interval 
of  16  months.  The  number  of  PRRs  conducted  ranged  from  a 
low  of  one  for  an  entire  program  to  a  high  of  six.  The 
program  which  planned  only  one  PRR  was  at  the  lower  end  on 
the  continuum  of  program  size,  cost,  and  complexity  while 
the  two  programs  which  planned  six  PRRs  were  at  the  higher 
end  of  this  continuum.  It  was  evident  from  this  review  that 
the  number  of  PRRs  planned  varied  in  proportion  to  the  size, 
cost,  and  complexity  of  the  weapon  system  program.  This  is 
further  reinforced  by  the  fact  that  the  size  of  the  weapon 
system  program  office  also  varied  in  proportion  to  the 
program's  size,  cost,  and  complexity.  Therefore,  it  follows 
that  the  smaller  programs  had  fewer  resources  to  commit  to 
PRR  planning,  implementation,  and  management. 

There  was  little  difference  between  the  average  number 
of  PRRs  planned  at  ASD  and  those  planned  at  MSD.  ASD  aver¬ 
aged  3-4  PRRs  planned  per  program  while  MSD  averaged  2-3 
PRRs  per  program.  This  difference  can  probably  be  attrib¬ 
uted  to  the  fact  that  the  programs  reviewed  at  ASD  were 
generally  much  larger  than  those  at  MSD. 

Second,  the  duration  of  the  PRRs  that  were  conducted 
also  varied  among  the  ten  programs.  The  average  duration  of 
all  PRRs  conducted  was  eight  days  with  a  range  of  three  days 
as  the  shortest  and  14  days  as  the  longest.  The  programs 
which  conducted  three  day  PRRs  were  on  the  lower  end  of  the 
continuum  of  program  size,  cost,  and  complexity  while  the 
programs  which  conducted  14  day  PRRs  were  at  the  higher  end. 
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Other  factors  which  determined  the  duration  of  PRRs  were  the 


contractor’s  performance  and  timing  of  the  PRRs  in  the  FSD 
phase.  In  several  cases,  if  the  contractor  was  considered 
responsive  and  had  proven  good  performance,  the  PRRs  were 
shorter.  Additionally,  the  PRRs  conducted  early  in  FSD  were 
usually  shorter  than  those  conducted  immediately  prior  to 
production . 

There  was  a  difference  of  only  one  day  in  the  average 
duration  of  PRRs  between  ASD  and  MSD .  The  duration  of  the 
PRR  was  the  actual  scheduled  review  time  and  did  not  include 
weekends  or  travel  time.  The  average  duration  of  PRRs  at 
MSD  was  approximately  seven  days  while  the  average  duration 
for  ASD  was  approximately  six  days.  This  difference  in 
duration  can  be  attributed  to  the  fact  that  more  PRRs  are 
conducted  per  program  at  ASD  than  at  MSD  resulting  in  more 
details  or  areas  to  cover  per  PRR  at  MSD.  This  fact  is 
further  reinforced  by  the  range  of  PRR  durations  at  both 
product  divisions  being  equal  and  varying  in  proportion  to 
the  same  program  size,  cost,  and  complexity  criteria.  ^ 

Third,  the  sizes  of  the  PRR  teams  also  varied  among 
the  ten  programs.  In  addition,  PRR  team  sizes  also  varied 
between  the  different  reviews  conducted  on  the  same  program. 
The  average  team  size  of  all  PRRs  conducted  was  36  people 
with  a  range  from  a  low  of  15  people  to  a  high  of  70  people. 
However,  the  three  largest  programs  had  teams  of  60  or  more 
people  and  inflated  the  average  for  all  ten  programs. 

By  disregarding  the  three  largest  programs,  the  average  team 
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size  was  27  people  with  a  range  of  15  people  for  a  low  and 
40  people  for  a  high. 

There  was  no  clear  factor  identifiable  which  deter¬ 
mined  the  sizes  of  PRR  teams.  As  expected,  the  three 
largest  programs  in  fact  had  the  largest  PRR  teams.  How¬ 
ever,  the  size  of  PRR  teams  for  the  remaining  programs  were 
not  in  proportion  to  program  size,  cost,  and  complexity. 

This  could  be  attributed  to  other  factors  that  might 
influence  team  size  such  as  PRR  duration  and  frequency. 

There  was  little  variation  in  the  sizes  of  PRR  teams 
at  ASD  and  those  at  MSD  although  ASD ' s  programs  were  much 
larger.  Actually,  PRRs  at  MSD  averaged  two  more  people  than 
those  at  ASD.  This  follows  the  generalization  that  PRR  team 
size  is  less  influenced  by  program  size,  cost,  and  complex¬ 
ity  but  rather  by  PRR  frequency  and  duration. 

Fourth,  the  average  number  of  review  panels  per  PRR 
for  all  ten  programs  was  six  with  a  range  of  three  panels 
for  a  low  to  eight  panels  for  a  high.  Only  one  program  used 
three  panels  while  the  rest  used  between  five  and  eight 
panels.  The  number  of  review  panels  did  not  vary  in  propor¬ 
tion  to  program  size  or  cost.  In  addition,  there  was  no 
panel  variation  between  the  different  PRRs  conducted  on  the 
same  program  indicating  the  timing  or  frequency  of  PRRs  was 
not  an  influence  on  the  number  of  panels  used.  It  was 
evident  that  the  primary  factor  influencing  the  number  of 
review  panels  was  the  complexity  of  the  weapon  system. 
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The  most  frequently  used  panels  among  the  ten  programs 
along  with  their  frequency  are  as  follows:  quality  assurance 
(10);  materi.il  t  (8);  manufacturing  (8);  engineer¬ 

ing  (7);  and  manufacturing  engineering  (6).  Any  additional 
panels  were  utilized  based  on  specific  program  needs  and 
weapon  system  complexity. 

There  was  no  difference  in  the  average  number  of 
panels  used  or  the  frequency  of  specific  panels  between  the 
programs  of  ASD  and  those  of  MSD.  This  reinforces  the 
generalization  that  the  number  of  panels  per  PRR  were  more 
influenced  by  program  complexity  than  by  program  size  and 
cost . 

Evaluation  criteria  and  risk  management  techniques 
among  the  ten  programs  will  be  addressed  in  a  later  investi¬ 
gative  question. 

Investigative  Question  2.  Is  there  adequate  guidance 
and  direction  provided  in  DOD  and  Air  Force  Directives  and 
Regulations  for  implementing  and  managing  PRRs? 

Both  DOD  Instruction  5000.38,  ‘Production  Readiness 
Reviews,"  and  Air  Force  Regulation  84-2,  ‘Production  Readi¬ 
ness  Reviews,'  were  referenced  in  the  PRR  Plans  for  all  ten 
programs.  Manufacturing  Directors  and  Manufacturing 
Officers  from  all  ten  programs  generally  felt  these 
regulations  provided  adequate  direction  for  the  basic  plan¬ 
ning  and  preparation  activities.  However,  actual  PRR 
implementation  and  management  rested  primarily  with  the 
experience  and  expertise  of  the  manufacturing  personnel 
involved  with  the  PRR.  This  generalization  is  reinforced  by 
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the  different  implementation  and  management  strategies 
employed  by  the  ten  programs  which  went  well  beyond  the 
direction  provided  in  the  regulations. 

Most  programs  utilized  the  regulations  for  basic  PRR 
planning  and  ensuring  that  the  program  met  both  DOD  and  Air 
Force  requirements.  In  addition  to  the  basic  requirements, 
PRRs  were  tailored  to  meet  the  program’s  needs.  These  needs 
were  primarily  based  on  program  size,  cost,  and  complexity. 
The  tailoring  of  PRRs  to  meet  program  needs  was  basically 
accomplished  through  the  expertise  and  experience  of  the 
program's  manufacturing  management  personnel. 

There  was  no  identifiable  difference  between  programs 
at  ASD  or  MSD  regarding  the  use  of  DOD  and  Air  Force 
Directives  in  planning  and  implementing  PRRs. 

Investigative  Question  3.  What  criteria,  policies, 
and  procedures  are  currently  used  by  the  different  program 
offices  in  evaluating  and  assessing  the  readiness  of  their 
weapon  system  and  contractor  to  begin  production? 

There  were  basically  three  different  approaches  used 
by  the  ten  programs  for  evaluating  and  assessing  production 
readiness  and  associated  risks.  The  three  approaches  used 
are  described  in  the  following  paragraphs. 

First,  the  two  largest  programs  implemented  nearly  all 
of  the  transition  templates  from  DOD  Manual  4245.7  in  their 
PRR  Plans.  The  extensive  template  approach  provided 
the  guidelines  for  which  the  entire  PRR  was  planned 
in  an  attempt  to  ensure  complete  reviews  of  these  large, 
complex  programs.  From  these  templates,  standards  were 
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developed  which  were  based  on  the  requirements  of  the 
contract  SOW  and  good  business  practices.  (See  Appendix  B 
for  a  sample  of  standards  developed  by  the  C-17  Program 
Office)  These  standards  provided  the  team  members  with  the 
following  review  data:  a  description  of  the  PRR  factor  being 
reviewed;  a  reference  to  any  applicable  MIL-STD  or  good 
business  practice;  a  set  of  subfactors  to  evaluate;  and  a 
list  of  data  required  for  review  to  adequately  assess  the 
PRR  factor.  In  addition,  each  team  member  was  given  a  copy 
of  DOD  Manual  4245.7  to  use  as  a  hand  guide. 

Risk  assessment  categories  were  also  taken  directly 
from  DOD  Manual  4245.7  to  determine  associated  production 
risks  from  the  PRR  findings.  These  risk  categories  were  as 
follows:  'high'  for  risks  likely  to  cause  major  impacts  to 
cost,  schedule,  or  performance;  ’medium"  for  risks  with 
potential  impacts;  and  "low"  for  risks  with  low  potential 
impacts . 

Production  readiness  concerns  and  risks  were  managed 
through  the  use  of  forms  entitled  Requests  For  Action  (RFA) . 
An  RFA  identifies  concerns  or  problems,  suggests  possible 
solutions,  and  requires  the  contractor  to  provide  a  correc¬ 
tive  action  plan  with  an  accompanying  schedule.  The  RFAs 
were  then  formally  tracked  at  subsequent  reviews.  The  second 
approach  had  three  programs  tailoring  the  use  of  the  transi¬ 
tion  templates  more  to  meet  specific  program  needs.  Speci¬ 
fic  evaluation  areas  were  identified  and  then  evaluation 
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criteria  or  questions  were  developed  by  team  members 
utilizing  applicable  templates.  These  criteria  or  questions 
generally  followed  the  guidelines  outlined  in  the  templates. 
However,  this  approach  relied  more  on  team  member  experience 
and  expertise.  In  addition,  each  team  member  was  given  a 
copy  of  DOD  Manual  4245.7  to  use  as  a  hand  guide. 

Risk  assessment  categories  for  these  programs  were 
taken  from  AFSC  Regulation  84-2.  The  risk  categories  for 
impacts  to  cost,  schedule,  and  performance  are  as  follows: 

" non-recoverable '  for  risks  which  will  cause  definite  sub¬ 
stantial  impact;  "high*  for  risks  where  adverse  impact  is 
likely;  "moderate"  for  risks  where  management  attention 
could  prevent  impacts;  and  "low"  when  there  is  no  evidence 
of  any  risk. 

Production  readiness  risks  or  concerns  were  managed 
for  all  three  programs  with  the  use  of  RFAs.  Two  programs 
had  formal  tracking  and  statusing  procedures  in  place  for 
managing  these  RFAs.  The  third  program  did  not  have  a 
formal  method  for  managing  RFAs  but  rather  left  the  respon¬ 
sibility  for  correction  totally  up  to  the  contractor. 

The  third  approach  combines  all  the  approaches  taken 
by  the  remaining  programs.  All  five  remaining  programs  used 
evaluation  criteria  that  was  a  compilation  of  previous  PRR 
Plans,  lessons  learned,  and  team  member  expertise.  In  most 
cases,  questions  were  generated  by  the  team  members  and 
approved  by  the  team  director.  This  approach  led  to  a  more 
tailored  PRR  based  on  progress  or  status  of  the  program. 
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Although  the  symbols  and  descriptions  were  slightly 
different,  it  was  apparent  that  all  five  programs  used  risk 
categories  based  on  AFSC  Regulation  84-2. 

Production  readiness  risks  or  concerns  were  managed  by 
three  of  the  five  programs  with  the  use  of  RFAs .  The 
primary  tracking  and  statusing  method  was  follow-on  incre¬ 
mental  reviews.  The  remaining  two  programs  did  not  have 
formal  management  procedures  for  risks  or  concerns.  Rather, 
this  responsibility  was  left  entirely  up  to  the  contractor. 
In  addition,  incremental  PRRs  were  used  to  verify  corrective 
actions . 

Investigative  Question  4.  Do  these  criteria, 
policies,  and  procedures  reflect  the  recommendations  of  both 
the  1985  GAO  study  and  the  1982  DSB  study  regarding  transi¬ 
tion  into  production? 

The  general  intent  of  both  studies  was  addressed  for 
incorporation  into  actual  PRR  implementation  and  management 
practices  of  all  ten  programs. 

First,  the  primary  thrust  or  focus  of  the  1982  DSB 
Task  Force  study  was  directed  towards  examining  ways  and 
methods  that  more  clearly  define  and  accelerate  the  transi¬ 
tion  from  development  to  production  of  weapon  systems.  In 
response  to  this  task,  a  matrix  of  the  most  critical  events 
in  the  design,  test,  and  production  elements  of  the  indus¬ 
trial  process  was  generated.  These  elements  were  used  to 
develop  a  set  of  templates  that  describe  techniques  for 
improving  the  acquisition  process  and  reducing  production 
risks.  These  templates  were  recommended  by  the  DSB  Task 
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Force  to  be  utilized  as  a  tool  for  transitioning  weapon 
systems  from  FSD  into  production. 

Five  of  the  ten  programs  made  extensive  use  of  these 
templates  in  planning,  implementing,  and  managing  PRRs .  The 
templates  were  tailored  to  meet  program  needs  and  were 
incorporated  in  the  program's  PRR  Plans.  In  addition,  all 
five  programs  provided  copies  of  DOD  Manual  4245.7  to  PRR 
team  members  prior  to  each  PRR. 

The  five  remaining  programs  did  not  use  templates  but 
rather  developed  their  own  evaluation  criteria  and  risk 
assessment  strategies.  Two  of  these  programs  did  not  incor¬ 
porate  the  recommendations  or  intent  of  the  DSB  Task  Force 
study.  Instead  of  identifying  and  minimizing  technical 
risks  during  transition,  a  minimal  number  of  PRRs  were 
conducted  which  took  a  'snapshot'  in  time  of  program 
progress.  The  other  three  programs  met  the  intent  of  the 
study  through  the  use  of  detailed  PRR  plans  and  evaluation 
criteria  which  paralleled  the  template  approach. 

Second,  the  emphasis  of  the  1985  GAO  study  was  to 
determine  why  some  weapon  systems  encounter  production 
problems  while  others  do  not.  The  primary  recommendations 
from  the  GAO  study  that  pertain  to  this  research  were  the 
employment  of  PRRs  as  a  tool  for  managing  production  prepar¬ 
ations  to  progressively  reduce  production  risks,  beginning 
early  and  repeating  them  at  intervals  during  FSD. 

Seven  of  the  ten  programs  had  detailed  plans  for 
multiple  PRRs  to  be  conducted  incrementally.  Each  of  these 


45 


seven  programs  planned  for  an  average  of  four  PRRs  with  an 
average  interval  of  16  months.  The  three  remaining  programs 
planned  for  an  average  1-2  PRRs  each  with  no  consistent 
interval.  It’s  apparent  that  the  seven  programs  with  the 
multiple  incremental  PRRs  meet  the  intent  of  the  GAO  recom¬ 
mendation  for  incremental  reviews. 

However,  the  average  time  interval  from  FSD  start-up 
to  the  initial  PRR  for  seven  programs  was  31  months.  The 
interval  for  the  remaining  three  programs  was  not  available. 
Although  the  time  interval  to  the  initial  PRR  is  dependent 
on  the  level  or  complexity  of  weapon  system  development,  it 
does  not  appear  that  the  intent  of  the  GAO  recommendation 
for  using  PRRs  early  in  FSD  is  being  met. 

Investigative  Question  5.  How  are  the  transition 
templates  from  DOD  Manual  4245.7,  'Transition  From  Develop¬ 
ment  To  Production,'  applied  to  PRR  implementation  and 
management  by  the  programs  which  used  them? 

The  five  programs  which  used  these  templates  had  two 
approaches.  First,  as  mentioned  previously,  the  two  largest 
programs  made  extensive  use  of  the  templates  with  a  detailed 
PRR  Plan  incorporating  a  majority  of  the  templates.  These 
programs  were  so  large  in  scope  and  complexity  that  nearly 
all  of  the  templates  were  applicable.  The  approach  taken  by 
the  remaining  three  programs  was  to  tailor  the  use  of  the 
templates  to  meet  specific  program  needs.  These  programs 
were  much  smaller  in  scope  and  complexity  and  thus,  in  the 
judgement  of  the  key  managers,  required  a  more  limited  use 
of  templates. 
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There  were  no  specific  templates  identified  which  were 
favored  by  any  PRR  teams.  In  addition  to  contractor  perfor¬ 
mance,  the  primary  influences  on  the  templates  used  were 
program  size,  cost,  and  complexity. 

Although  five  programs  made  extensive  use  of  the 
templates,  these  programs  were  provided  little  guidance  or 
direction.  The  Manufacturing  Directors  for  three  of  these 
programs  were  introduced  to  the  use  of  templates  at  a  three 
day  Navy  conference.  The  remaining  two  Manufacturing 
Directors  incorporated  the  use  of  these  templates  at  the 
suggestion  of  HQ  AFSC .  All  five  Manufacturing  Directors 
stated  they  would  continue  to  use  the  templates  on  future 
reviews . 

Investigative  Question  6.  What  contractual  require¬ 
ments  were  placed  on  the  contractor  to  plan  for  and  support 
PRRs  ? 

Very  few  requirements  were  outlined  in  the  contract 
SOW  for  each  of  the  ten  programs.  The  basic  reference  in 
each  program’s  SOW  called  for  the  contractors  to  participate 
in  PRRs.  Several  of  the  contracts  referenced  a  requirement 
for  the  contractor  to  perform  PRRs  at  the  major  subcontrac¬ 
tors.  There  was  only  one  program  which  made  reference  to 
more  specific  planning  criteria  and  more  detailed  contractor 
support  requirements.  However,  these  criteria  were  still 
lacking  enough  detail  for  contractual  enforcement. 

Investigative  Question  7.  Of  what  perceived  value  are 
PRRs  to  program  managers  in  transitioning  their  programs 
into  production? 
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Only  five  program  managers  were  available  for  inter¬ 
view  on  this  subject.  Three  of  these  program  managers  felt 
the  PRRs  were  adequately  conducted  and  provided  the  required 
information  to  help  transition  their  programs.  Of  the 
remaining  two  program  managers,  one  felt  a  need  for  addi¬ 
tional  data  while  the  other  did  not  like  the  PRR  concept  and 
felt  it  was  inadequate. 

The  role  the  PRRs  played  in  helping  program  managers 
transition  their  programs  into  production  parallels  the 
adequacy  breakdown  mentioned  above.  Three  program  managers 
felt  the  PRRs  played  a  very  important  role  in  managing  their 
programs.  This  was  reinforced  by  the  fact  that  these 
program  managers  actively  planned  and  participated  in  their 
PRRs.  In  addition,  the  Manufacturing  Directors  for  these 
programs  felt  their  program  managers  were  totally  committed 
to  the  PRR  process.  The  remaining  two  program  managers  were 
unsure  of  the  role  of  PRRs  in  their  programs  and  were  not 
committed  to  the  PRR  process.  The  Manufacturing  Directors 
for  these  two  programs  had  total  responsibility  for  planning 
and  implementing  their  PRRs. 

Summary 

This  chapter  detailed  the  findings  of  the  study  in  the 
form  of  responses  to  the  investigative  questions.  These 
findings  were  based  both  on  a  detailed  review  of  PRR 
documentation  and  informal  interviews  of  program  managers 
and  manufacturing  directors.  The  next  and  final  chapter 
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presents  conclusions  and  recommendations  with  regard  to 
actual  PRR  implementation  and  management  practices. 
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I V .  Conclusions  and  Recommendations 


This  chapter  presents  the  conclusions  and  associated 
recommendations  of  this  research.  The  conclusions  drawn  are 
based  on  data  from  the  literature  review  and  case  study 
findings  of  actual  PRR  implementation.  The  recommendations 
are  based  on  one  or  more  of  the  following:  program 
specific  PRR  practices  that  improved  PRR  management;  changes 
to  current  PRR  practices  that  could  further  improve  PRR 
management;  and  suggested  changes  taken  from  the  interviews 
of  program  managers  and  manufacturing  managers. 

Conclusions 

The  primary  objective  of  this  research  was  to  deter¬ 
mine  how  PRRs  are  planned  and  implemented  during  the 
transition  from  development  to  production  of  selected  weapon 
systems  within  AFSC .  Ba3ed  on  the  findings  presented  in 
Chapter  III,  several  conclusions  were  drawn. 

First,  it  was  evident  that  each  of  the  ten  programs 
followed  the  basic  DOD  and  Air  Force  requirements  for 
conducting  PRRs.  All  ten  programs  conducted  at  least  one 
PRR  prior  to  the  milestone  III  decision.  However,  both  the 
DOD  and  Air  Force  Directives  lacked  sufficient  guidance  for 
implementing  and  managing  PRRs.  There  were  no  criteria 
available  to  help  program  offices  determine  the  most 
beneficial  approach  to  help  transition  their  programs  into 
production.  Once  the  basic  requirements  for  planning  PRRs 
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were  met,  actual  PRR  implementation  and  management  relied 
heavily  upon  previous  experience  and  lessons  learned,  when 
available.  In  many  cases,  PRR  plans  were  taken  from  other 
programs  and  used  while  a  few  programs  extensively  planned 
their  own  PRRs .  This  approach,  or  policy,  of  leaving 
implementation  and  management  up  to  the  individual  program 
offices  has  both  positive  and  negative  implications. 

A  positive  aspect  of  this  approach  was  that  program 
offices  were  allowed  to  tailor  the  whole  PRR  process  to  meet 
their  specific  program  needs.  In  most  cases,  the  PRR 
process  was  tailored  by  developing  program  specific  evalua¬ 
tion  criteria;  adjusting  the  number  and  frequency  of  PRRs 
conducted;  and  adjusting  PRR  team  size.  This  approach 
allowed  more  flexibility  in  managing  weapon  system  programs 
through  the  transition  process.  However,  this  approach 
relied  heavily  on  PRR  team  member  expertise  and  experience. 

Conversely  a  negative  aspect  was  that  the  smaller 
programs  did  not  have  the  resources  to  extensively  plan  and 
implement  PRRs.  Without  the  expertise  and  experience, 
several  smaller  programs  were  forced  not  to  tailor  the 
PRR  process  to  meet  program  needs  but  rather  just  meet  the 
basic  PRR  planning  requirements  identified  in  DOD  and  Air 
Force  Directives.  Two  programs  in  particular,  did  not  have 
a  plan  for  managing  areas  of  concern  or  deficiencies 
identified  by  the  PRR  team.  In  these  two  cases,  without 
specific  directions  or  guidance,  the  PRR  process  was  not 
adequately  implemented  and  managed.  With  this  approach,  the 


opportunity  for  a  program  to  enter  into  production  without 
adequate  preparation  is  increased  and  could  increase  the 
likelihood  of  cost  overruns  and  schedule  delays. 

All  ten  programs  identified  basic  and  general  require¬ 
ments  and  objectives  in  the  contract  statement  of  work 
(SOW).  Although  the  SOW  is  the  primary  vehicle  for  the 
government  to  bind  the  contractor  to  his  responsibility, 
none  of  the  programs  identified  specific  PRR  objectives, 
requirements,  and  schedules.  The  basic  requirements  in  the 
SOWs  were  to  conduct  PRRs  on  the  prime  contractors  and  the 
requirement  for  the  prime  contractors  to  conduct  PRRs  on  the 
major  subcontractors.  The  specific  requirements  for 
contractor  support  and  preparation  for  PRRs  was  levied 
informally.  This  approach  was  generally  successful. 

However,  two  programs  having  non-respons i ve  contractors  had 
ineffective  management  of  PRR  areas  of  concern  and  deficien¬ 
cies.  In  addition,  the  contractors  were  ill  prepared  for 
the  initial  PRRs  on  their  programs. 

The  PRR  Plans  identified  program  specific  PRR  imple¬ 
mentation  and  management  strategies.  Usually  included  in 
the  PRR  Plan  were  PRR  objectives,  evaluation  criteria,  team 
structure,  and  review  schedules.  All  ten  programs  had 
detailed  PRR  Plans.  However,  two  programs  did  not  have 
evaluation  criteria  included  in  their  PRR  Plans.  Instead, 
team  members  generated  questions  to  include  in  the  PRR  and 
submitted  them  to  the  team  leaders  prior  to  the  start  of 
each  PRR.  Most  programs  did  not  change  the  PRR  Plan  between 
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incremental  reviews.  The  manufacturing  directors  were 
responsible  for  development  and  management  of  PRR  Plans  for 
their  programs . 

The  number  of  PRRs  varied  in  proportion  to  the  size, 
cost,  and  complexity  of  the  weapon  system  program.  However, 
excluding  the  two  larger  programs,  the  duration  of  PRRs 
appeared  to  vary  inversely  to  the  number  of  PRRs  conducted. 
The  smaller,  less  complex  programs  conducted  fewer  PRRs  with 
longer  durations  while  the  larger  programs  conducted  more 
PRRs  with  shorter  durations. 

There  was  no  clear  factor  identified  during  the  review 
that  influenced  the  size  of  the  PRR.  Some  of  the  smaller 
programs  had  larger  PRR  teams  than  much  larger  programs.  It 

appears  that  PRR  team  size  was  a  variable  that  may  have  been 

used  to  compensate  for  other  factors  affecting  PRRs  such  as 
duration  and  frequency.  Smaller  programs  may  have  enlisted 
larger  teams  to  help  ensure  all  areas  of  production 
readiness  were  reviewed  in  a  lesser  number  of  reviews. 

The  PRR  panel  structure  used  by  each  program  was  not 
influenced  by  program  size  or  cost.  Rather,  the  primary 
factor  influencing  panel  structure  was  weapon  system 
complexity.  The  more  complex  systems  required  more  review 

panels  while  the  less  complex  systems  required  fewer. 

PRR  management  was  reviewed  for  risk  assessment  and 
risk  management  techniques.  All  ten  programs  had  risk 
assessment  categories  identified  in  their  PRR  Plans.  How¬ 
ever,  the  risk  assessment  process  was  very  subjective  in 
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nature  and  not  quantifiable.  Although  the  categories 
was  identified,  actual  classification  of  findings  or 
concerns  were  essentially  left  to  the  PRR  Panel  Chiefs  or 
Team  Director.  The  five  programs  which  used  the  DOD  Manual 
4245.7  transition  templates  had  more  specific  criteria  to 
assist  in  classifying  findings.  However,  the  process  still 
relied  on  subjective  decisions  from  the  PRR  team  leaders. 

Risk  management  was  generally  accomplished  with  the 
use  of  requests  for  action  and  follow-up  reviews.  Seven  of 
the  programs  used  this  method  of  requesting  corrective 
actions  from  contractors  for  any  concerns  or  deficiencies. 
These  corrective  actions  or  plans  were  then  tracked  and 
monitored  during  follow-up  reviews.  The  three  remaining 
programs  had  no  formal  risk  management  process  but  rather 
left  total  responsibility  to  the  contractor  for  corrective 
actions . 

A  second  research  objective  was  to  determine  if  the 
recommendations  from  the  1985  GAO  study  (Why  Some  Programs 
Encounter  Production  Problems  While  Others  Do  Not)  were 
implemented.  The  primary  recommendations  as  they  pertain  to 
this  research  were  to  ensure  that  an  adequate  level  of 
production  preparations  were  conducted  early  and  continuous¬ 
ly  throughout  FSD.  In  addition,  PRRs  should  be  used  as  a 
tool  for  managing  these  production  preparations  to  progress¬ 
ively  reduce  production  risks,  beginning  early  and  repeating 
them  at  specified  intervals  during  FSD. 
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These  recommendations  were  not  fully  implemented  on 
any  of  the  ten  programs.  Seven  programs  planned  for  incre¬ 
mental  PRRs  throughout  FSD.  However,  the  average  time  lapse 
from  the  beginning  of  FSD  to  the  initial  PRR  was  31  months. 
The  remaining  three  programs  did  not  plan  incremental  PRRs 
but  rather  planned  for  a  minimum  number  of  reviews  closer  to 
the  Milestone  III  decision. 

The  final  research  objective  was  to  review  how  the 
transition  templates  from  the  1982  DSB  Task  Force  study  on 
transition  were  being  applied  in  the  PRR  process.  Five 
programs  actually  used  the  transition  templates  for  PRR 
planning  and  implementation.  There  were  two  basic 
approaches  used  to  incorporate  the  templates  into  the  PRR 
Plan . 

One  approach  that  was  taken  by  two  larger  programs  was 
the  implementation  of  nearly  all  of  the  templates.  This 
extensive  approach  provided  the  guidelines  for  which  the 
entire  PRR  process  was  planned  to  ensure  a  complete  review 
of  these  complex  programs.  Included  in  this  approach  were 
the  risk  assessment  criteria  taken  directly  from  the  tem¬ 
plates  . 

The  second  approach  involved  three  programs  that 
tailored  the  use  of  the  transition  templates  to  meet  specif¬ 
ic  program  needs.  These  programs  were  smaller  and  did  not 
require  the  all  encompassing  template  approach.  Specific 
evaluation  areas  were  identified  and  then  evaluation 
criteria  or  questions  were  developed  by  team  members 
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utilizing  applicable  templates.  These  questions  generally 
followed  the  guidelines  outlined  by  the  templates.  However, 
this  approach  relied  more  heavily  on  team  member  expertise 
and  experience. 

Recommendations 

In  response  to  the  conclusions  listed  above,  and  in  an 
attempt  to  improve  the  PRR  process,  a  number  of  recommenda¬ 
tions  are  presented  below.  These  recommendations  also 
reflect  suggestions  taken  from  the  interviews  of  program 
managers  and  manufacturing  managers. 

Recommendation  Number  One.  The  DOD  and  Air  Force 
Directives  and  Regulations  governing  PRR  implementation  and 
management  should  be  revised  to  provide  more  guidance  and 
direction  for  planning  PRRs . 

General  guidelines  including  proven  PRR  techniques 
should  be  provided  that  correlate  to  program  size,  cost,  and 
complexity.  This  research  reflects  the  significance  of  this 
criteria  on  the  PRR  process.  The  additional  guidance  and 
directions  could  help  program  offices  better  design  and 
tailor  a  PRR  process  to  meet  specific  program  needs.  The 
PRR  is  probably  as  important  to  the  production  phase  as  the 
PDR  and  CDR  are  to  the  FSD  phase.  Perhaps  developing  a  MIL- 
STD  for  PRRs  similar  to  MIL-STD  1521B  for  design  reviews 
could  provide  the  necessary  PRR  planning  guidance.  With 
this  guidance,  smaller  programs  with  limited  manufacturing 
resources  might  be  compelled  to  better  plan  PRRs  to  meet 
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their  program  needs  rather  than  simply  meeting  basic  DOD 
requirements . 

Recommendation  Number  Two.  PRR  objectives  and 
requirements  should  be  specified  in  greater  detail  in  the 
contract  SOW.  The  SOW  is  the  primary  vehicle  the  government 
uses  to  levy  contractual  requirements  and  support  for  PRRs 
on  the  contractor.  However,  there  were  few  specific 
requirements  identified  in  the  SOWs  for  any  of  the  programs 
reviewed.  The  SOW  should  specify  the  purpose  and  goals  of 
the  PRRs.  In  addition,  a  tentative  schedule  should  be 
included  to  allow  both  the  government  and  contractor  to 
adequately  prepare  for  these  reviews.  Also,  specific 
requirements  for  preparation  and  support  of  PRRs  is  needed 
to  ensure  that  contractors  understand  their  roles  and 
responsibilities.  Citing  regulatory  guidance  and  identi¬ 
fying  review  techniques  could  also  improve  contractor 
preparations . 

Recommendation  Number  Three.  Provide  more  training 
and  preparation  for  PRR  team  members  prior  to  their  respec¬ 
tive  PRRs.  Several  program  managers  and  manufacturing 
managers  voiced  the  need  for  more  experienced  personnel  for 
PRRs.  Due  to  the  shortage  of  experienced  manufacturing 
personnel,  training  existing  personnel  and  improving  PRR 
oreparations  are  the  best  approach  to  fulfill  this 
recommendation.  In  addition,  better  use  of  CAS  personnel 
administering  the  program  in  the  field  coul  orovide 
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insights  into  production  deficiencies  that  might  not 
otherwise  be  detected  during  a  short  review. 

Recommendation  Number  Four.  Program  offices  should 
conduct  joint  government  and  contractor  PRR  planning  meet¬ 
ings  at  least  60  days  prior  to  each  PRR.  This  joint  meeting 
could  provide  the  contractor  a  better  direction  or  focus  for 
preparing  for  specific  PRRs .  The  object  of  these  PRRs  is  to 
prepare  the  contractor  and  the  weapon  system  for  production. 
By  providing  the  contractor  specific  PRR  objectives  and 
evaluation  criteria,  preparation  for  the  PRR  and,  more 
importantly,  the  transition  to  production  should  be 
improved.  In  addition,  specific  roles  and  responsibilities 
for  both  the  government  and  the  contractor  can  be  reviewed 
for  clarification  or  change. 

Recommendation  for  Further  Research 

This  research  revealed  several  different  approaches  to 
planning,  implementing,  and  managing  PRRs.  These  different 
approaches  reflected  a  multitude  of  techniques  based  primar¬ 
ily  on  program  size,  cost,  and  complexity.  In  addition,  the 
level  of  experience  and  expertise  of  manufacturing  personnel 
also  contributed  to  the  particular  approaches  used. 

Further  research  could  help  answer  the  following 
questions  in  determining  the  best  PRR  approaches.  How 
successful  will  these  weapon  system  programs  be  when  they 
enter  the  production  phase?  Which  approach  for  planning, 
implementing,  and  managing  PRRs  best  prepared  programs  for 
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production?  Did  the  programs  which  used  the  transition 
templates  fare  well  in  production?  Was  the  use  of  the 
transition  templates  a  significant  contributor  to  program 
success? 

These  questions  could  be  answered  by  conducting  a 
follow-on  study  in  approximately  24-36  months.  These  ten 
programs  will  be  well  into  production  and  would  provide  a 
basis  for  correlating  their  success  with  the  approach  taken 
for  production  preparations.  With  this  further  study, 
recommendations  for  continued  PRR  improvement  could  be  made. 
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Appendix  A:  Case  Study  Questions 


Program  Background 

1.  What  is  the  weapon  system  being  procured"?  (Contract) 


2.  Which  product  division  is  responsible  for  managing  this 
acquisi tion? 


3.  When  did  FSD  begin?  (Contract) 


4.  What  was  the  dollar  value  of  the  FSD  contract"? 


5.  When  was  the  production  decision  made"?  (Program 
Manager,  Contract) 


6.  What  is  the  dollar  value  of  the  production  contract"? 
(Contract) 


7.  What  is  the  primary  deliverable  end  item  and  quantity 
to  be  produced?  (Contract) 


60 


PRR  Requirements  in  Contract  Statement  of  Work 


1.  What  is  the  dollar  amount  identified  in  the  contract 
for  PRR  planning  and  implementation?  (Contract) 


2.  What  are  the  PRR  requirements  identified  in  the  SOW? 


3.  When  is  PRR  planning  to  begin?  (SOW) 


4.  How  many  PRRs  were  planned?  (SOW,  PRR  Plan) 


5.  What  contractor  involvement  in  PRRs  is  identified  in 
the  SOW? 


6.  What  additional  government  or  contractor  requirements 
are  identified  in  the  SOW  to  support  PRR  planning  and 
implementation? 
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PRR  Plan 


1.  How  many  PRRs  were  conducted?  (PRR  Plan,  PRR  Report) 


2.  When  was  the  first  PRR  conducted  with  respect  to  the 
beginning  of  FSD?  (PRR  Plan,  Contract) 


3.  If  more  than  one  PRR  was  conducted,  were  they  conducted 
incremental lj ?  (PRR  Plan) 


4.  When  was  the  last  PRR  conducted  with  respect  to  the 
production  decision?  (PRR  Plan) 


5.  What  were  the  stated  objectives  of  the  PRRs?  (PRR 
Plan,  In-briefs,  Policy) 


6.  What  aws  the  size  of  each  PRR  team?  (PRR  Plan,  In- 
brief) 


7.  What  was  the  duration  of  each  PRR?  (PRR  Plan,  PRR 
Report) 


8.  What  type  of  CAS  support  was  provided?  (PRR  Report) 


9.  What  type  of  contractor  support  was  provided?  (PRR 
Report) 


10.  Were  templates  from  DOD  Manual  4245.7  used  in 
evaluating  production  readiness  and  risks?  (PRR  Plan,  In¬ 
brief) 
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11.  If  these  templates  were  not  used,  what  evaluation 
criteria  was  used  in  determining  production  readiness  and 
risk  assessments?  (PRR  Plan,  In-brief) 


12.  If  DOD  Manual  4245.7  was  used,  which  templates  were 
used  in  the  evaluation?  (PRR  Plan,  In-brief) 


13.  How  were  production  readiness  concerns  or  risks  managed 
prior  to  start  of  production’  (PRR  Plan,  PRR  Report) 


63 


Program  Manager  Interview 


1.  Did  the  PRR  process  and  report  provide  you  with 
adequate  production  readiness  and  risk  data  to  assist  you  in 
managing  the  transition  of  your  program  into  production? 


2.  What  additional  production  readiness  data,  if  any,  did 
you  need  to  manage  your  program'? 


3.  Did  the  PRR  process  play  a  role  in  the  management  of 
your  program’s  transition  into  production7 


4.  Are  you  familiar  with  DOD  Manual  4245.7  'Transition 
From  Development  into  production? 


5.  What  changes  would  you  suggest  to  improve  the  PRR 
process? 
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Manufacturing  Manager  Interview 


1.  Do  you  feel  there  is  a  commitment  from  the  program 
office  for  planning  and  implementing  PRRs? 


2.  Do  you  feel  that  PRRs  are  adequately  planned  and 
implemented  for  your  program? 


3.  Do  you  feel  that  the  contractor  has  adequately  prepared 
for  and  supported  PRRs? 


4.  What  changes  in  the  PRR  process  would  you  recommend? 


5.  Are  you  familiar  with  DOD  Manual  4245.7  "Transition 
From  Development  to  Production?" 


6.  Have  you  ever  used  DOD  Manual  4245.7  as  a  guide  for 
PRRs?  If  so,  would  you  use  this  approach  again?  (i.e.  are 
the  templates  valuable?) 


7.  If  you  have  used  DOD  Manual  4245.7  during  PRRs,  where 
did  you  get  direction  or  guidance  for  their  use? 


8.  Which  templates  did  you  use? 


9.  Are  there  any  templates  which  you  feel  significantly 
aided  in  evaluating  and  assessing  the  readiness  of  the 
program  for  production? 
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Appendix  B: 


Sample  Evaluation  Standards 


Panel  1.0  Manufacturing 

Factor  1.1  Manufacturing  Management  and  Support 

Subfactor  1.1.4  Manufacturing  Management  Systems 

Reference:  SOW  1 062JAA,  MIL-STD-1 528 

Description:  This  subfactor  involves  the  assessment  of 

existing/planned  system  for  managing  the  manufacturing  effort.  Specific 
topics  to  be  evaluated  are  description  of  management  systems,  systems  uti¬ 
lization,  effects  of  concurrency,  and  management  systems  interaction  and 

> 

relationship  with  other  disciplines.  This  subfactor  also  assesses  the 
contractor's  total  manufacturing  organization  and  how  it  integrates  with 
other  functional  disciplines.  Specific  topics  to  be  highlighted  are  manu¬ 
facturing  integration,  organization  and  concurrency. 

The  subfactor  is  satisfied  when  the  following  standards  are  adequately  met: 

1  .  The  contractor  shall  have  and  use  a  manufacturing  management  system 
-hat  implements  MIL-STD-1 528  (SOW  1062JAA).  This  is  verified  by  the 
f ollcwing: 

a.  AF  approved  Manufacturing  Plan 

b.  Adequate  and  complete  procedures 

c.  Procedures  that  are  current  and  controlled 

d.  Compliance  with  procedures  by  intended  users 
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2.  Clear  definition  of  authority  and  responsibilities  of  the  organizations 
and  personnel  charged  with  execution  of  the  manufacturing  management 
system. 

3.  Internal  review  process  to  monitor  manufacturing  management  system 
effectiveness. 

4.  h  complete,  detailed  manufacturing  organiza¬ 

tion  chart  with  the  lines  of  authority  and  responsibilities  clearly  defined. 

5.  Clear  definition  of  vertical  and  horizontal  lines  of  communication  and 

interfaces  with  other  (Quality,  Engineering,  Material, 

etc . ) . 
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Panel  2.0  Engineering 


Factor  2.2  Producibility  of  Design 

Description:  This  factor  examines  the  contractor's  progress  in  imple¬ 

menting  their  plans  and  procedures  in  terms  of  ensuring  that  manufacturing 
engineering  has  continuing  input  to  design  engineering  during  product 
design  and  documentation  activities. 

This  factor  is  satisfied  when  the  following  standards  are  adequately 
met: 


1.  The  contractor  follows  their  objectives,  policies,  and  procedures 
to  enhance  producibility  based  on  past  experience,  accepted  commercial 
practices,  and  Air  Force/.  lessons  learned. 

2.  The  defined  interface  relationships  between  Manufacturing 
Producibility  (MPO)  and  Engineering  Producibility  (EPO)  organizations  is 
producing  verifiable  exchanges  and  incorporation  of  changes. 

3.  The  contractor  is  abiding  by  the  policies  and  procedures  to  stan¬ 
dardize  the  number  and  types  of  materials,  processes,  and  parts  and  is  able 
to  demonstrate  progress  in  this  area. 

4.  The  contractor  has  resolved  design  approach  conflicts  between  manu¬ 
facturing  and  product  design  functions. 

5.  The  contractor  can  provide  examples  of  producibility  concerns  being 
addressed  by  design  engineering  and  other  functions  such  as  logistics  and 
manu  factur ing . 
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6.  The  contractor  can  provide  any  changes  to  policies,  procedures,  or 
plans  since  the  last  PRR  and  the  impact  of  these  changes  to  the  producibi- 
lity  function. 

7.  The  contractor  has  control  of  subcontractor  producibility  efforts. 
Appropriate  requirements  are  incorporated  in  subcontracts  and  are  being 
monitored. 


8.  Producibility  studies  have  been  conducted  and  are  utilized  by 
design/manufacturing  engineering.  These  studies  include  interfaces  with 
other  areas  such  as  Logistics  and  Manufacturing. 

9.  Producibility  engineering  is  reviewing  all  drawings  in  accordance 
with  established  procedures  and  in  a  timely  manner.  The  drawing  review 
cycle  allows  time  for  producibility  review. 

10.  Training  in  producibility  is  conducted  on  a  continuing  basis  for 
all  engineering  disciplines. 

11.  Employee  suggestion/shop  feedback  is  actively  pursued  and  result 
implemented  where  appropriate. 

12.  The  producibility  function  is  staffed  to  adequately  handle  present 
workloads.  A  manloading  plan  is  in  effect  to  cover  future  needs. 
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Panel  4.0  Quality  Assurance 

Factor  4.1  Quality  Program 

Description:  This  factor  evaluates  the  contractors  plans,  procedures  and  docu¬ 
mentation  used  to  conduct  an  effective,  economical  and  integrated  quality 
assurance  program  that  includes  provisions  for  assuring  quality  in  design,  deve¬ 
lopment,  fabrication,  processing,  assembly,  test  and  preparation  for  shipment. 

Subfactor  4.1.1  Organization  and  Planning 

Description:  This  subfactor  evaluates  the  contractor's  management  of  the 

quality  program. 

This  subfactor  is  satisfied  when  the  following  standards  are  adequately 
met; 


1  .  The  contractor's  quality  program  follows  the  Quality  Program  Plan  (P410). 
Identifies  how  the  Quality  Program  Plan  is  utilized/and  maintained  by  the 
contractor . 

2.  Each  of  the  quality  requirements  within  the  SOW  is  specifically 

addressed  and  adequately  covered  by  the  Quality  Program. 

(MIL-Q-9858A,  3.2, 

3.  Personnel  performing  quality  functions  have  been  identified  and  have 
well  defined  responsibility,  authority,  and  organizational  freedom  to  iden¬ 
tify  and  solve  quality  problems.  (MIL-Q-9858A,  3.1) 

4.  The  quality  program  provides  for  analysis  and  use  of  records  as  a  basis 
for  management  action.  Adequate  evidence  exists  to  show  that  management 
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action  considers  quality  program  inputs.  Quality  reports  are  furnished  to 
management  for  information  and  action.  (MIL-Q-98S3A,  3.4,  DAC  CP  10.021, 

Regulation  9) 

5.  The  contractor  identifies,  maintains  and  uses  specific  cost  data  to 
identify  the  cost  of  prevention,  detection  and  correction  of  nonconforming 
supplies.  The  contractor  maintains  surveillance  of  program  quality  costs. 
(MIL-Q-9858A,  3.6, 

6.  The  contractor  is  maintaining  a  Quality  Improvement  Program  compatible  with 
the  ASD  Quality  Improvement  Strategy .  The  contractor's  program  has  a  baseline 
against  which  tangible  improvement  can  be  shown. 
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PANEL  7.0  TEST/SAFETY 


FACTOR  7 . 2  SAFETY 

SUBFACTOR  7.2.3  System  Safety  Program 

Description:  This  subfactor  evaluates  the  System  Safety  Program  to 

ensure  the  contractor  has  developed  and  implemented  adequate,  aggres¬ 
sive  measures,  in  accordance  with  the  contract-,  to  identify,  evaluate 
and  eliminate  or  control  hazards  throughout  the  design  of  the 
system. 

The  subfactor  is  satisfied  when  the  following  standards  are  met: 

1 .  The  contractor  has  developed,  and  updated  as  required,  an  adequate 
System  Safety  Program  Plan  which  describes  in  detail  the  efforts  to 
be  implemented  to  comply  with  the  contractual  system  safety  require¬ 
ments  . 

2  .  The  contractor  is  adhering  to  the  schedule  of  events  contained  in 
the  System  Safety  Program  Plan. 

3.  The  contractor  presents  adequate  evidence  to  indicate  that  the 
System  Safety  Office  interfaces  with  each  of  the  subsystem  engineering 
divisions  to  ensure  an  integrated  safety  effort  is  accomplished. 

4  .  The  contractor  shows  they  are  conducting  rigorous  hazard  analyses 
in  accordance  with  the  contract,  to  identify,  evaluate  and  eliminate  or 
control  system  hazards. 

5.  The  contractor  has  developed  and  implemented  a  hazard  tracking 
system  that  ensures  all  category  I  and  II  hazards,  and  those  with  an 
RHI  of  8  or  greater,  are  monitored  to  ensure  corrective  actions  are 
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incorporated  into  the  design.  The  contractor  must  also  show  this 
tracking  system  is  accurate  and  up-to-date. 

6.  The  contractor  shows  that  no  identified  hazard  remains  with  an  RHI 
of  12  or  greater. 

7.  The  contractor  demonstrates  that  they  are  making  maximum  use  of  the 
LSA  data  base  as  it  becomes  available . 

8  .  The  contractor  shows  active  interfacing  with  the  test  engineers  to 
ensure  the  appropriate  safety  requirements  are  incorporated  into  the 
plans,  procedures,  work  efforts  and  results  for  all  testing  activities. 

9.  The  contractor  demonstrates  that  thorough  system  safety  programs 
are  being  conducted  by  their  subcontractors  and  suppliers. 

10.  The  contractor  is  making  active  contact  with  the  support  equipment 
engineers  to  influence  the  system  safety  and  occupational  safety  of 
newly  developed  or  significantly  modified  support  equipment. 
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PANEL  8.0  FINANCIAL  MANAGEMENT 


FACTOR  8 . 1  BUDGET  EXECUTION 

SUBFACTOR  8.1.2  Contract  Funds  Status 

Description:  This  subfactor  addresses  the  requirements  of  the  Contract 
Funds  Status  Report  (CFSR)  prepared  in  compliance  with  the  Contract 
Data  Requirements  List  (CDRL) ,  SEQ#301B.  The  CFSR  will  supply  complete 
and  timely  funding  data  for  the  contract  to  the  system  manager. 

Delivered  data  will  be  in  compliance  with  the  Data  Item  Description 
(DID)  to  assure  the  adequate  detail  necessary  for;  (a)  updating  and 
forecasting  contract  funds  requirements,  (b)  planning  and  decision 
making  on  funding  changes  in  the  contract,  (c)  developing  fund  require¬ 
ments  and  budget  estimates  in  support  of  the  approved  program,  and  (d) 
obtaining  rough  estimates  of  the  contract  termination  costs.  CFSR  fore¬ 
casts  will  reflect  good  business/financial  management  practices  and 
procedures . 

This  subfactor  is  satisfied  when  the  following  standards  are  adequately 
met : 

1.  Contractual  Requirement  for  Contract  Funds  Status  Report  (CFSR) 

a.  The  CFSR  is  prepared  and  submitted  to  the  Air  Force  25  calendar 
days  after  the  close  of  the  quarterly  accounting  period. 

b.  The  required  number  of  copies  are  provided  for  Air  Force 
distribution . 

c.  Costs  are  segregated  by  Cost  Plus  ’Incentive  Fee  (CPIF)  and 
Fixed  Price  (FP)  contract  line  items  at  the  Work  Breakdown  Struc¬ 
ture  (WBS)  1  level. 

d.  cumulative  time  periods  are  reported  properly  by  month  for  6 
months,  quarterly  to  the  end  of  the  current  government  fiscal  year. 
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The  data  item  description  requirements  (DI-F-6004B) 


2  . 


a.  The  initial  negotiated  contract  target  price  and  contract 
ceiling  price  (item  9)  include  only  the  share  associated  with  the 
current  fiscal  year,  (item  3) . 

b.  The  adjusted  contract  price  and  ceiling  (item  10)  include  only 
the  share  associated  with  current  fiscal  year  (item  3) . 

c.  The  funding  authorized  to  date  (col  C)  includes  the  dollar 
amounts  under  the  contract  from  the  beginning  of  the  contract 
through  the  current  report  date  (item  6) . 

d.  The  "accrued  expenditures  plus  open  commitments  total"  (col  D) 
includes  accrued  expenditures  through  the  end  of  the  reporting 
period  and  open  commitments  on  the  as-of-date  of  the  report . 

e.  The  "contract  work  authorized-def initizea"  (col  E)  includes  the 
estimated  price  of  the  negotiated  contract  for  the  current  fiscal 
year  and  only  the  contract  changes  which  have  been  priced  and 
incorporated  in  the  contract. 

f.  The  "contract  work  author ized-not  definitized"  (col  F)  includes 
only  that  work  for  which  written  orders  have  been  received. 

g.  The  "forecast-not  yet  authorized"  includes  the  estimate  for 
changes  and  proposals,  including  fee.  The  narrative  remarks  sec¬ 
tion  includes  all  change  documents  and  the  estimated  value  of  each. 

h.  The  "open  commitments"  line  includes  accrued  expenditures. 

i.  The  accrued  expenditures  line  includes: 

(1)  Actual  payments  for  services  or  items 
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^This  study  reviewed  ten  programs  which  recently 
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briefings.  In  addition,  key  managers  were  interviewed  for 
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